Files
microdao-daarion/site/tasks/TASK_PHASE_NODE_SELF_HEALING_v1/index.html
Apple ef3473db21 snapshot: NODE1 production state 2026-02-09
Complete snapshot of /opt/microdao-daarion/ from NODE1 (144.76.224.179).
This represents the actual running production code that has diverged
significantly from the previous main branch.

Key changes from old main:
- Gateway (http_api.py): expanded from ~40KB to 164KB with full agent support
- Router: new /v1/agents/{id}/infer endpoint with vision + DeepSeek routing
- Behavior Policy: SOWA v2.2 (3-level: FULL/ACK/SILENT)
- Agent Registry: config/agent_registry.yml as single source of truth
- 13 agents configured (was 3)
- Memory service integration
- CrewAI teams and roles

Excluded from snapshot: venv/, .env, data/, backups, .tgz archives

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-09 08:46:46 -08:00

1188 lines
43 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!doctype html>
<html lang="en" class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<link rel="canonical" href="https://IvanTytar.github.io/microdao-daarion/tasks/TASK_PHASE_NODE_SELF_HEALING_v1/">
<link rel="icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.5.3, mkdocs-material-9.5.18">
<title>TASK_PHASE_NODE_SELF_HEALING_v1 - DAARION Documentation</title>
<link rel="stylesheet" href="../../assets/stylesheets/main.66ac8b77.min.css">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce((e,_)=>(e<<5)-e+_.charCodeAt(0),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
</head>
<body dir="ltr">
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
<label class="md-overlay" for="__drawer"></label>
<div data-md-component="skip">
<a href="#task_phase_node_self_healing_v1" class="md-skip">
Skip to content
</a>
</div>
<div data-md-component="announce">
</div>
<header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="DAARION Documentation" class="md-header__button md-logo" aria-label="DAARION Documentation" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54Z"/></svg>
</a>
<label class="md-header__button md-icon" for="__drawer">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2Z"/></svg>
</label>
<div class="md-header__title" data-md-component="header-title">
<div class="md-header__ellipsis">
<div class="md-header__topic">
<span class="md-ellipsis">
DAARION Documentation
</span>
</div>
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
TASK_PHASE_NODE_SELF_HEALING_v1
</span>
</div>
</div>
</div>
<script>var media,input,key,value,palette=__md_get("__palette");if(palette&&palette.color){"(prefers-color-scheme)"===palette.color.media&&(media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']"),palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent"));for([key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script>
<label class="md-header__button md-icon" for="__search">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5Z"/></svg>
</label>
<div class="md-search" data-md-component="search" role="dialog">
<label class="md-search__overlay" for="__search"></label>
<div class="md-search__inner" role="search">
<form class="md-search__form" name="search">
<input type="text" class="md-search__input" name="query" aria-label="Search" placeholder="Search" autocapitalize="off" autocorrect="off" autocomplete="off" spellcheck="false" data-md-component="search-query" required>
<label class="md-search__icon md-icon" for="__search">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M9.5 3A6.5 6.5 0 0 1 16 9.5c0 1.61-.59 3.09-1.56 4.23l.27.27h.79l5 5-1.5 1.5-5-5v-.79l-.27-.27A6.516 6.516 0 0 1 9.5 16 6.5 6.5 0 0 1 3 9.5 6.5 6.5 0 0 1 9.5 3m0 2C7 5 5 7 5 9.5S7 14 9.5 14 14 12 14 9.5 12 5 9.5 5Z"/></svg>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12Z"/></svg>
</label>
<nav class="md-search__options" aria-label="Search">
<button type="reset" class="md-search__icon md-icon" title="Clear" aria-label="Clear" tabindex="-1">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M19 6.41 17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12 19 6.41Z"/></svg>
</button>
</nav>
</form>
<div class="md-search__output">
<div class="md-search__scrollwrap" data-md-scrollfix>
<div class="md-search-result" data-md-component="search-result">
<div class="md-search-result__meta">
Initializing search
</div>
<ol class="md-search-result__list" role="presentation"></ol>
</div>
</div>
</div>
</div>
</div>
</nav>
</header>
<div class="md-container" data-md-component="container">
<main class="md-main" data-md-component="main">
<div class="md-main__inner md-grid">
<div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href="../.." title="DAARION Documentation" class="md-nav__button md-logo" aria-label="DAARION Documentation" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54Z"/></svg>
</a>
DAARION Documentation
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../public/" class="md-nav__link">
<span class="md-ellipsis">
Home
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../public/getting-started/" class="md-nav__link">
<span class="md-ellipsis">
Getting Started
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../public/architecture-overview/" class="md-nav__link">
<span class="md-ellipsis">
Architecture
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../public/daiS_daos_overview/" class="md-nav__link">
<span class="md-ellipsis">
DAIS & DAOS
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--section md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="">
<span class="md-ellipsis">
Internal
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
Internal
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5_1" >
<label class="md-nav__link" for="__nav_5_1" id="__nav_5_1_label" tabindex="0">
<span class="md-ellipsis">
Infra
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_5_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5_1">
<span class="md-nav__icon md-icon"></span>
Infra
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../internal/infra/INFRA_AUTOMATION_PACK_V1/" class="md-nav__link">
<span class="md-ellipsis">
Infra Automation Pack v1
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../internal/infra/monitoring_overview/" class="md-nav__link">
<span class="md-ellipsis">
Monitoring Overview
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../internal/infra/nodes_registry_v0/" class="md-nav__link">
<span class="md-ellipsis">
Nodes Registry v0
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5_2" >
<label class="md-nav__link" for="__nav_5_2" id="__nav_5_2_label" tabindex="0">
<span class="md-ellipsis">
Specs
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_5_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5_2">
<span class="md-nav__icon md-icon"></span>
Specs
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../internal/specs/matrix_presence_aggregator/" class="md-nav__link">
<span class="md-ellipsis">
Matrix Presence Aggregator
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../internal/specs/city_map_spec/" class="md-nav__link">
<span class="md-ellipsis">
City Map Spec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../internal/specs/node_join_protocol_draft/" class="md-nav__link">
<span class="md-ellipsis">
Node Join Protocol (Draft)
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-sidebar md-sidebar--secondary" data-md-component="sidebar" data-md-type="toc" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#_1" class="md-nav__link">
<span class="md-ellipsis">
Проєкт
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#_2" class="md-nav__link">
<span class="md-ellipsis">
Фаза
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#_3" class="md-nav__link">
<span class="md-ellipsis">
Статус
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#_4" class="md-nav__link">
<span class="md-ellipsis">
Мета
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#problem-statement" class="md-nav__link">
<span class="md-ellipsis">
Problem Statement
</span>
</a>
<nav class="md-nav" aria-label="Problem Statement">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#_5" class="md-nav__link">
<span class="md-ellipsis">
Симптом
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#_6" class="md-nav__link">
<span class="md-ellipsis">
Причини
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#_7" class="md-nav__link">
<span class="md-ellipsis">
Рішення
</span>
</a>
<nav class="md-nav" aria-label="Рішення">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#1-node-registry" class="md-nav__link">
<span class="md-ellipsis">
1. Node Registry — єдине джерело істини
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#2-self-registration-api" class="md-nav__link">
<span class="md-ellipsis">
2. Self-Registration API
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#3-node-bootstrap-script" class="md-nav__link">
<span class="md-ellipsis">
3. Node Bootstrap Script
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#4-node-guardian-self-healing-loop" class="md-nav__link">
<span class="md-ellipsis">
4. Node Guardian Self-Healing Loop
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#_8" class="md-nav__link">
<span class="md-ellipsis">
Файли
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#self-healing" class="md-nav__link">
<span class="md-ellipsis">
Інваріанти Self-Healing
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#_9" class="md-nav__link">
<span class="md-ellipsis">
Використання
</span>
</a>
<nav class="md-nav" aria-label="Використання">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#_10" class="md-nav__link">
<span class="md-ellipsis">
При першому деплої ноди
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#guardian-loop" class="md-nav__link">
<span class="md-ellipsis">
Запуск Guardian Loop
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#docker-compose" class="md-nav__link">
<span class="md-ellipsis">
Через Docker Compose
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#self-healing_1" class="md-nav__link">
<span class="md-ellipsis">
Self-Healing сценарії
</span>
</a>
<nav class="md-nav" aria-label="Self-Healing сценарії">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#1-directory" class="md-nav__link">
<span class="md-ellipsis">
Сценарій 1: Нода зникла з Directory після деплою
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#2-heartbeat" class="md-nav__link">
<span class="md-ellipsis">
Сценарій 2: Heartbeat застарів
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#3-agent-count-0" class="md-nav__link">
<span class="md-ellipsis">
Сценарій 3: Agent count = 0
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#acceptance-criteria" class="md-nav__link">
<span class="md-ellipsis">
Acceptance Criteria
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#_11" class="md-nav__link">
<span class="md-ellipsis">
Наступні кроки
</span>
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-content" data-md-component="content">
<article class="md-content__inner md-typeset">
<h1 id="task_phase_node_self_healing_v1">TASK_PHASE_NODE_SELF_HEALING_v1<a class="headerlink" href="#task_phase_node_self_healing_v1" title="Permanent link">&para;</a></h1>
<h2 id="_1">Проєкт<a class="headerlink" href="#_1" title="Permanent link">&para;</a></h2>
<p>DAARION.city — Nodes / Node Cabinet / DAGI Router</p>
<h2 id="_2">Фаза<a class="headerlink" href="#_2" title="Permanent link">&para;</a></h2>
<p>Self-healing нод (автоматична реєстрація, відновлення та синхронізація)</p>
<h2 id="_3">Статус<a class="headerlink" href="#_3" title="Permanent link">&para;</a></h2>
<p><strong>COMPLETED</strong></p>
<hr />
<h2 id="_4">Мета<a class="headerlink" href="#_4" title="Permanent link">&para;</a></h2>
<p>Зробити так, щоб:</p>
<ol>
<li>Ноди <strong>ніколи не "зникали"</strong> з Node Directory, якщо фізично існують і шлють heartbeat</li>
<li>Реєстрація/оновлення нод виконувалась <strong>агентами ноди</strong>, а не ручними діями</li>
<li>Node Directory → Node Cabinet → Node Metrics → DAGI Router були повністю узгоджені</li>
</ol>
<hr />
<h2 id="problem-statement">Problem Statement<a class="headerlink" href="#problem-statement" title="Permanent link">&para;</a></h2>
<h3 id="_5">Симптом<a class="headerlink" href="#_5" title="Permanent link">&para;</a></h3>
<ul>
<li><code>/nodes</code> (Node Directory) показує:</li>
<li>«Знайдено нод: 0»</li>
<li>«Помилка завантаження нод»</li>
<li>Хоча:</li>
<li>насправді NODE1/NODE2 є в <code>node_cache</code></li>
<li>метрики, DAGI Router, агенти ноди працюють</li>
</ul>
<h3 id="_6">Причини<a class="headerlink" href="#_6" title="Permanent link">&para;</a></h3>
<ul>
<li>Node Directory фронт дивився на іншу структуру даних</li>
<li>Реєстрація ноди не відпрацьовувала після деплою</li>
<li>Немає самовідновлюваної логіки на рівні нод</li>
</ul>
<hr />
<h2 id="_7">Рішення<a class="headerlink" href="#_7" title="Permanent link">&para;</a></h2>
<h3 id="1-node-registry">1. Node Registry — єдине джерело істини<a class="headerlink" href="#1-node-registry" title="Permanent link">&para;</a></h3>
<p><strong>Таблиця:</strong> <code>node_registry</code></p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">node_registry</span><span class="w"> </span><span class="p">(</span>
<span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="p">,</span><span class="w"> </span><span class="c1">-- node_id</span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"> </span><span class="c1">-- Людська назва</span>
<span class="w"> </span><span class="n">hostname</span><span class="w"> </span><span class="nb">text</span><span class="p">,</span><span class="w"> </span><span class="c1">-- Hostname</span>
<span class="w"> </span><span class="n">environment</span><span class="w"> </span><span class="nb">text</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"> </span><span class="c1">-- production/development/staging</span>
<span class="w"> </span><span class="n">roles</span><span class="w"> </span><span class="nb">text</span><span class="p">[]</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="s1">&#39;{}&#39;</span><span class="p">,</span><span class="w"> </span><span class="c1">-- [&#39;gpu&#39;, &#39;ai_runtime&#39;, ...]</span>
<span class="w"> </span><span class="n">description</span><span class="w"> </span><span class="nb">text</span><span class="p">,</span>
<span class="w"> </span><span class="n">is_active</span><span class="w"> </span><span class="nb">boolean</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="k">true</span><span class="p">,</span>
<span class="w"> </span><span class="n">registered_at</span><span class="w"> </span><span class="n">timestamptz</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">now</span><span class="p">(),</span>
<span class="w"> </span><span class="n">updated_at</span><span class="w"> </span><span class="n">timestamptz</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">now</span><span class="p">(),</span>
<span class="w"> </span><span class="n">last_self_registration</span><span class="w"> </span><span class="n">timestamptz</span><span class="p">,</span><span class="w"> </span><span class="c1">-- Остання самореєстрація</span>
<span class="w"> </span><span class="n">self_registration_count</span><span class="w"> </span><span class="nb">integer</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="mi">0</span>
<span class="p">);</span>
</code></pre></div>
<p><strong>View для Node Directory:</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">VIEW</span><span class="w"> </span><span class="n">v_nodes_directory</span><span class="w"> </span><span class="k">AS</span>
<span class="k">SELECT</span><span class="w"> </span>
<span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="o">*</span><span class="p">,</span>
<span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">cpu_model</span><span class="p">,</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">gpu_model</span><span class="p">,</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">ram_total</span><span class="p">,</span><span class="w"> </span><span class="p">...</span>
<span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">last_heartbeat</span><span class="p">,</span>
<span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">agent_count_router</span><span class="p">,</span>
<span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">agent_count_system</span><span class="p">,</span>
<span class="w"> </span><span class="k">CASE</span><span class="w"> </span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">last_heartbeat</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">NOW</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">INTERVAL</span><span class="w"> </span><span class="s1">&#39;10 minutes&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;stale&#39;</span>
<span class="w"> </span><span class="k">ELSE</span><span class="w"> </span><span class="s1">&#39;online&#39;</span>
<span class="w"> </span><span class="k">END</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">connection_status</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">node_registry</span><span class="w"> </span><span class="n">r</span>
<span class="k">LEFT</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">node_cache</span><span class="w"> </span><span class="k">c</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">node_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">id</span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">r</span><span class="p">.</span><span class="n">is_active</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">true</span><span class="p">;</span>
</code></pre></div>
<h3 id="2-self-registration-api">2. Self-Registration API<a class="headerlink" href="#2-self-registration-api" title="Permanent link">&para;</a></h3>
<table>
<thead>
<tr>
<th>Endpoint</th>
<th>Метод</th>
<th>Опис</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>/internal/nodes/register-or-update</code></td>
<td>POST</td>
<td>Самореєстрація ноди</td>
</tr>
<tr>
<td><code>/internal/node/{node_id}/heartbeat</code></td>
<td>POST</td>
<td>Heartbeat з метриками</td>
</tr>
<tr>
<td><code>/internal/node/{node_id}/directory-check</code></td>
<td>GET</td>
<td>Перевірка видимості</td>
</tr>
<tr>
<td><code>/internal/node/{node_id}/self-healing/status</code></td>
<td>GET</td>
<td>Статус self-healing</td>
</tr>
<tr>
<td><code>/internal/node/{node_id}/self-healing/trigger</code></td>
<td>POST</td>
<td>Тригер self-healing</td>
</tr>
<tr>
<td><code>/internal/nodes/needing-healing</code></td>
<td>GET</td>
<td>Список нод для healing</td>
</tr>
</tbody>
</table>
<h3 id="3-node-bootstrap-script">3. Node Bootstrap Script<a class="headerlink" href="#3-node-bootstrap-script" title="Permanent link">&para;</a></h3>
<p><strong>Файл:</strong> <code>scripts/node-bootstrap.sh</code></p>
<div class="codehilite"><pre><span></span><code><span class="c1"># Використання при старті ноди</span>
<span class="nv">NODE_ID</span><span class="o">=</span>node-2-macbook-m4max<span class="w"> </span><span class="se">\</span>
<span class="nv">NODE_NAME</span><span class="o">=</span><span class="s2">&quot;MacBook Pro M4 Max&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="nv">NODE_ENVIRONMENT</span><span class="o">=</span>development<span class="w"> </span><span class="se">\</span>
<span class="nv">NODE_ROLES</span><span class="o">=</span>gpu,ai_runtime,development<span class="w"> </span><span class="se">\</span>
./scripts/node-bootstrap.sh
</code></pre></div>
<p><strong>Що робить:</strong>
1. Відправляє POST на <code>/internal/nodes/register-or-update</code>
2. При успіху — відправляє початковий heartbeat
3. При помилці — retry до 5 разів</p>
<h3 id="4-node-guardian-self-healing-loop">4. Node Guardian Self-Healing Loop<a class="headerlink" href="#4-node-guardian-self-healing-loop" title="Permanent link">&para;</a></h3>
<p><strong>Файл:</strong> <code>scripts/node-guardian-loop.py</code></p>
<div class="codehilite"><pre><span></span><code><span class="c1"># Запуск як фоновий процес</span>
<span class="nv">NODE_ID</span><span class="o">=</span>node-2-macbook-m4max<span class="w"> </span><span class="se">\</span>
<span class="nv">NODE_NAME</span><span class="o">=</span><span class="s2">&quot;NODE2&quot;</span><span class="w"> </span><span class="se">\</span>
python<span class="w"> </span>scripts/node-guardian-loop.py<span class="w"> </span>--interval<span class="w"> </span><span class="m">60</span>
<span class="c1"># Одноразова перевірка</span>
python<span class="w"> </span>scripts/node-guardian-loop.py<span class="w"> </span>--node-id<span class="w"> </span>node-2-macbook-m4max<span class="w"> </span>--once
</code></pre></div>
<p><strong>Що перевіряє:</strong>
1. Чи нода видима в Node Directory
2. Чи є heartbeat
3. Чи є Guardian/Steward агенти
4. Чи є агенти в router</p>
<p><strong>Self-healing дії:</strong>
1. Якщо не видима — виконує self-registration
2. Якщо heartbeat старий — відправляє новий
3. Якщо статус error — тригерить healing через API</p>
<hr />
<h2 id="_8">Файли<a class="headerlink" href="#_8" title="Permanent link">&para;</a></h2>
<table>
<thead>
<tr>
<th>Файл</th>
<th>Опис</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>migrations/039_node_registry_self_healing.sql</code></td>
<td>Міграція для node_registry</td>
</tr>
<tr>
<td><code>services/city-service/repo_city.py</code></td>
<td>Функції для self-healing</td>
</tr>
<tr>
<td><code>services/city-service/routes_city.py</code></td>
<td>API endpoints</td>
</tr>
<tr>
<td><code>scripts/node-bootstrap.sh</code></td>
<td>Bootstrap скрипт</td>
</tr>
<tr>
<td><code>scripts/node-guardian-loop.py</code></td>
<td>Self-healing loop</td>
</tr>
</tbody>
</table>
<hr />
<h2 id="self-healing">Інваріанти Self-Healing<a class="headerlink" href="#self-healing" title="Permanent link">&para;</a></h2>
<table>
<thead>
<tr>
<th>Умова</th>
<th>Дія</th>
</tr>
</thead>
<tbody>
<tr>
<td>Нода не в node_registry</td>
<td>→ self_register()</td>
</tr>
<tr>
<td>heartbeat &gt; 10 хв</td>
<td>→ send_heartbeat()</td>
</tr>
<tr>
<td>agent_count_router = 0</td>
<td>→ alert + try reinstall</td>
</tr>
<tr>
<td>guardian_agent_id = NULL</td>
<td>→ alert</td>
</tr>
<tr>
<td>self_healing_status = error</td>
<td>→ trigger_healing()</td>
</tr>
</tbody>
</table>
<hr />
<h2 id="_9">Використання<a class="headerlink" href="#_9" title="Permanent link">&para;</a></h2>
<h3 id="_10">При першому деплої ноди<a class="headerlink" href="#_10" title="Permanent link">&para;</a></h3>
<div class="codehilite"><pre><span></span><code><span class="c1"># 1. Запустити міграцію</span>
psql<span class="w"> </span>-d<span class="w"> </span>daarion<span class="w"> </span>&lt;<span class="w"> </span>migrations/039_node_registry_self_healing.sql
<span class="c1"># 2. Запустити bootstrap</span>
<span class="nv">NODE_ID</span><span class="o">=</span>node-2-macbook-m4max<span class="w"> </span><span class="se">\</span>
<span class="nv">NODE_NAME</span><span class="o">=</span><span class="s2">&quot;MacBook Pro M4 Max&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="nv">NODE_ENVIRONMENT</span><span class="o">=</span>development<span class="w"> </span><span class="se">\</span>
./scripts/node-bootstrap.sh
</code></pre></div>
<h3 id="guardian-loop">Запуск Guardian Loop<a class="headerlink" href="#guardian-loop" title="Permanent link">&para;</a></h3>
<div class="codehilite"><pre><span></span><code><span class="c1"># Через systemd</span>
<span class="o">[</span>Unit<span class="o">]</span>
<span class="nv">Description</span><span class="o">=</span>DAARION<span class="w"> </span>Node<span class="w"> </span>Guardian
<span class="nv">After</span><span class="o">=</span>network.target
<span class="o">[</span>Service<span class="o">]</span>
<span class="nv">Environment</span><span class="o">=</span><span class="nv">NODE_ID</span><span class="o">=</span>node-2-macbook-m4max
<span class="nv">Environment</span><span class="o">=</span><span class="nv">NODE_NAME</span><span class="o">=</span>NODE2
<span class="nv">Environment</span><span class="o">=</span><span class="nv">CITY_SERVICE_URL</span><span class="o">=</span>http://localhost:7001
<span class="nv">ExecStart</span><span class="o">=</span>/usr/bin/python3<span class="w"> </span>/path/to/scripts/node-guardian-loop.py
<span class="nv">Restart</span><span class="o">=</span>always
<span class="o">[</span>Install<span class="o">]</span>
<span class="nv">WantedBy</span><span class="o">=</span>multi-user.target
</code></pre></div>
<h3 id="docker-compose">Через Docker Compose<a class="headerlink" href="#docker-compose" title="Permanent link">&para;</a></h3>
<div class="codehilite"><pre><span></span><code><span class="nt">services</span><span class="p">:</span>
<span class="w"> </span><span class="nt">node-guardian</span><span class="p">:</span>
<span class="w"> </span><span class="nt">image</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python:3.11-slim</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NODE_ID=node-2-macbook-m4max</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">NODE_NAME=NODE2</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">CITY_SERVICE_URL=http://city-service:7001</span>
<span class="w"> </span><span class="nt">command</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">python /app/scripts/node-guardian-loop.py</span>
<span class="w"> </span><span class="nt">volumes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">./scripts:/app/scripts</span>
<span class="w"> </span><span class="nt">depends_on</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">city-service</span>
</code></pre></div>
<hr />
<h2 id="self-healing_1">Self-Healing сценарії<a class="headerlink" href="#self-healing_1" title="Permanent link">&para;</a></h2>
<h3 id="1-directory">Сценарій 1: Нода зникла з Directory після деплою<a class="headerlink" href="#1-directory" title="Permanent link">&para;</a></h3>
<div class="codehilite"><pre><span></span><code><span class="mf">1.</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="n">Guardian</span><span class="w"> </span><span class="n">запускається</span>
<span class="mf">2.</span><span class="w"> </span><span class="n">check_visibility</span><span class="p">()</span><span class="w"> </span><span class="err"></span><span class="w"> </span><span class="n">false</span>
<span class="mf">3.</span><span class="w"> </span><span class="n">self_register</span><span class="p">()</span><span class="w"> </span><span class="err"></span><span class="w"> </span><span class="n">успіх</span>
<span class="mf">4.</span><span class="w"> </span><span class="n">check_visibility</span><span class="p">()</span><span class="w"> </span><span class="err"></span><span class="w"> </span><span class="n">true</span>
<span class="mf">5.</span><span class="w"> </span><span class="err"></span><span class="w"> </span><span class="n">Нода</span><span class="w"> </span><span class="n">знову</span><span class="w"> </span><span class="n">в</span><span class="w"> </span><span class="n">Directory</span>
</code></pre></div>
<h3 id="2-heartbeat">Сценарій 2: Heartbeat застарів<a class="headerlink" href="#2-heartbeat" title="Permanent link">&para;</a></h3>
<div class="codehilite"><pre><span></span><code><span class="mf">1.</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="n">Guardian</span><span class="w"> </span><span class="n">перевіряє</span><span class="w"> </span><span class="n">статус</span>
<span class="mf">2.</span><span class="w"> </span><span class="n">self_healing_status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;stale_heartbeat&quot;</span>
<span class="mf">3.</span><span class="w"> </span><span class="n">send_heartbeat</span><span class="p">()</span><span class="w"> </span><span class="err"></span><span class="w"> </span><span class="n">успіх</span>
<span class="mf">4.</span><span class="w"> </span><span class="err"></span><span class="w"> </span><span class="n">Heartbeat</span><span class="w"> </span><span class="n">оновлено</span>
</code></pre></div>
<h3 id="3-agent-count-0">Сценарій 3: Agent count = 0<a class="headerlink" href="#3-agent-count-0" title="Permanent link">&para;</a></h3>
<div class="codehilite"><pre><span></span><code><span class="mf">1.</span><span class="w"> </span><span class="n">Node</span><span class="w"> </span><span class="n">Guardian</span><span class="w"> </span><span class="n">бачить</span><span class="w"> </span><span class="n">agent_count_router</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mf">0</span>
<span class="mf">2.</span><span class="w"> </span><span class="n">Логує</span><span class="w"> </span><span class="n">попередження</span>
<span class="mf">3.</span><span class="w"> </span><span class="p">(</span><span class="n">Опційно</span><span class="p">)</span><span class="w"> </span><span class="n">trigger_healing</span><span class="p">()</span><span class="w"> </span><span class="n">для</span><span class="w"> </span><span class="n">перевірки</span><span class="w"> </span><span class="n">DAGI</span><span class="w"> </span><span class="n">Router</span>
<span class="mf">4.</span><span class="w"> </span><span class="err">⚠️</span><span class="w"> </span><span class="n">Потребує</span><span class="w"> </span><span class="n">уваги</span><span class="w"> </span><span class="n">адміністратора</span>
</code></pre></div>
<hr />
<h2 id="acceptance-criteria">Acceptance Criteria<a class="headerlink" href="#acceptance-criteria" title="Permanent link">&para;</a></h2>
<table>
<thead>
<tr>
<th>Критерій</th>
<th>Статус</th>
</tr>
</thead>
<tbody>
<tr>
<td>node_registry таблиця створена</td>
<td></td>
</tr>
<tr>
<td>API self-registration працює</td>
<td></td>
</tr>
<tr>
<td>node-bootstrap.sh виконує реєстрацію</td>
<td></td>
</tr>
<tr>
<td>node-guardian-loop.py запускається</td>
<td></td>
</tr>
<tr>
<td>Ноди видимі в /nodes після реєстрації</td>
<td></td>
</tr>
<tr>
<td>Self-healing при зникненні</td>
<td></td>
</tr>
<tr>
<td>Heartbeat оновлює статус</td>
<td></td>
</tr>
</tbody>
</table>
<hr />
<h2 id="_11">Наступні кроки<a class="headerlink" href="#_11" title="Permanent link">&para;</a></h2>
<ol>
<li><strong>Автоматичний DAGI Router reinstall</strong> при <code>agent_count_router = 0</code></li>
<li><strong>NATS events</strong> для node healing (<code>node.selfhealing.*</code>)</li>
<li><strong>Prometheus metrics</strong> для self-healing</li>
<li><strong>Alert rules</strong> для критичних станів</li>
<li><strong>Node Federation</strong> — з'єднання нод між собою</li>
</ol>
</article>
</div>
<script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script>
</div>
</main>
<footer class="md-footer">
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-copyright">
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
</div>
</div>
</div>
</footer>
</div>
<div class="md-dialog" data-md-component="dialog">
<div class="md-dialog__inner md-typeset"></div>
</div>
<script id="__config" type="application/json">{"base": "../..", "features": ["navigation.sections", "navigation.instant", "content.code.copy"], "search": "../../assets/javascripts/workers/search.b8dbb3d2.min.js", "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}}</script>
<script src="../../assets/javascripts/bundle.3220b9d7.min.js"></script>
</body>
</html>