[Feature] 引擎重启兜底策略：容器不重启重拉引擎 + 自杀裁决收敛 Daemon-MindIE-Motor-AtomGit

M.agent/skills/motor-dev/references/code-style.md+1-1

	@@ -132,7 +132,7 @@ The hook also removes now-unused `from typing import ...` lines. `Any` remains t
132		132
133	### ThreadSafeSingleton Pattern	133	### ThreadSafeSingleton Pattern
134		134
135	-Used by all core components (InstanceManager, FaultManager, EngineManager, HeartbeatManager, MetricsCollector):	135	+Used by all core components (InstanceManager, FaultManager, RegisterManager, HeartbeatManager, MetricsCollector):
136		136
137	```python	137	```python
138	class MyComponent(ThreadSafeSingleton):	138	class MyComponent(ThreadSafeSingleton):

M.agent/skills/motor-dev/references/nodeman.md+132-39

	@@ -11,31 +11,61 @@ Controller heartbeats.
11	NodeManager (Application)	11	NodeManager (Application)
12	│	12	│
13	├── NodeManagerAPI (FastAPI thread)	13	├── NodeManagerAPI (FastAPI thread)
14	-│ POST /node-manager/start — validate StartCmdMsg and launch native engines	14	+│ POST /node-manager/start — spawn engines with StartCmdMsg
15	-│ POST /node-manager/stop — stop native process groups	15	+│ POST /node-manager/stop — kill engines + delayed SIGTERM self (exit -1 →
16	-│ POST /node-manager/pause — mark endpoints PAUSED for PreStop	16	+│ k8s pod restart) — the "suicide" instruction
17	-│ POST /node-manager/resume — restore PAUSED endpoints	17	+│ POST /node-manager/engine-restart — relaunch engines in place, no pod restart
18	-│ GET /node-manager/status — report endpoint readiness	18	+│ body {"action": "restart"\|"abort", "instance_id"?}
19	-│ GET /readiness — Kubernetes readiness	19	+│ (thin route: restart delegates the whole
		20	+│ relaunch to Daemon.restart_engine, maps
		21	+│ its errors to 400/409/500;
		22	+│ abort = unfreeze suicide → heartbeat fallback)
		23	+│ POST /node-manager/pause — pause endpoints (snapshot/upgrade flow)
		24	+│ POST /node-manager/resume — resume endpoints
		25	+│ GET /node-manager/status — current engine states (relaxed=true for relaunch poll)
		26	+│ GET /readiness — k8s readiness probe
		27	+│ (TLS via mgmt_tls_config)
20	│	28	│
21	-├── Daemon	29	+├── RegisterManager (ThreadSafeSingleton)
22	-│ service registry for engine and optional KV-store services	30	+│ Registration protocol: POST /controller/register (with retry)
23	-│ 5-second service monitor → health_check()	31	+│ Ranktable file writing: saves ranktable JSON for engine RPC
		32	+│ Persists master_dp_ip/role/endpoints for engine relaunch (get_restart_params)
24	│	33	│
25	-├── NativeEngineService	34	+├── Daemon (ThreadSafeSingleton)
26	-│ builds LaunchContext, selects Native Engine Backend, delegates lifecycle to ProcessSupervisor	35	+│ Service registry orchestration (core/services/registry.py):
		36	+│ "engine" services → EngineService (subprocess.Popen, device pinning)
		37	+│ "kv-store" backends → memcache/lifecycle services
		38	+│ Device pinning via ASCEND_RT_VISIBLE_DEVICES env var
		39	+│ SIGKILL on stop (no graceful shutdown — engines are stateless)
		40	+│ 5s process monitor thread → svc.health_check() for every service
		41	+│ (PID death → freeze suicide + report ENGINE_DEAD to Controller)
		42	+│ 3s suicide-arbitration thread (single pod-rescheduling decision point):
		43	+│ 5 consecutive ABNORMAL observations after the grace period → suicide flag;
		44	+│ freeze window (deadline-based) suspends counting; only endpoints that
		45	+│ were NORMAL before are reported as dead (cold-start guard)
		46	+│ restart_engine(instance_id): owns the whole relaunch — serializes
		47	+│ (in-progress flag, EngineRestartInProgressError on overlap), resolves
		48	+│ launch params from RegisterManager (EngineRestartParamError on
		49	+│ missing/mismatch), freezes suicide (unfreezes on failure), pauses the
		50	+│ FaultReporter for the relaunch window and resumes after (fresh engines
		51	+│ restart their startup grace)
		52	+│ FaultReporter (owned here, third monitoring source): started in
		53	+│ pull_engine, stopped in stop(), (re)configured in update_config
27	│	54	│
28	-├── ProcessSupervisor	55	+├── HeartbeatManager (ThreadSafeSingleton)
29	-│ subprocess.Popen(start_new_session=True)	56	+│ Two daemon threads:
30	-│ owns RuntimeProcess records, process groups and native health probes	57	+│ _engine_server_status_thread — poll each engine GET /status every interval
31	-│	58	+│ _heartbeat_report_thread — POST /controller/heartbeat every interval
32	-├── HeartbeatManager	59	+│ Endpoint-state facts only (status polling + heartbeat reporting) —
33	-│ polls ProcessSupervisor every second	60	+│ arbitration lives in the Daemon, not here
34	-│ reports Controller heartbeat at configured interval	61	+│ No engine-readiness logic: status probing waits for the Daemon's
35	-│ preserves STARTING/STOPPING/PAUSED semantics and suicide threshold	62	+│ engine-ready handoff (mgmt ports up), injected at start()
36	│	63	│
37	└── FaultReporter	64	└── FaultReporter
38	- optional GET {business_port}/fault_tolerance/status polling for engine software faults	65	+ HTTP poll GET {business_port}/fault_tolerance/status per engine
		66	+ (vLLM FT REST API) → report_software_fault to Controller
		67	+ pause()/resume(): suspended by the Daemon across an engine relaunch —
		68	+ resume() clears poll state so re-pulled engines get a fresh startup grace
39	```	69	```
40		70
41	`motor/node_manager/core/services/native_engine/` is the native engine service boundary. Its	71	`motor/node_manager/core/services/native_engine/` is the native engine service boundary. Its
	@@ -48,11 +78,35 @@ depend on these node-local runtime states.
48	### Phase 1: Startup and Registration	78	### Phase 1: Startup and Registration
49		79
50	``` text	80	``` text
51	-main.py	81	+
52	- → NodeManagerConfig.from_json()	82	+1. main.py is a thin wrapper (41 lines): load config → port setup → NodeManager(config).run()
53	- → port allocation / configuration validation	83	+
54	- → NodeManager(Application).run()	84	+ The real orchestration lives in NodeManager(Application) (motor/node_manager/node_manager.py)
55	- → init_modules(): Daemon, NodeManagerAPI, EngineManager, HeartbeatManager	85	+ and the Application base class (motor/common/app/application.py): signal handlers,
		86	+ daemon tick loop, module init/start, graceful shutdown. There is no separate
		87	+ suicide_procedure() function — suicide is driven by the tick loop (see Phase 4).
		88	+
		89	+2. NodeManager.init_modules() — registration order:
		90	+ Daemon → NodeManagerAPI → [RegisterManager → HeartbeatManager]
		91	+
		92	+ RegisterManager/HeartbeatManager are registered ONLY when daemon.has_engine
		93	+ is True. A KV-only pod (kv_cache_store_config mode="separated") registers
		94	+ just Daemon + NodeManagerAPI.
		95	+
		96	+3. RegisterManager._register() [background thread]
		97	+
		98	+ Loop:
		99	+ wait_until_api_ready(timeout=30.0) # NodeManagerAPI must be serving
		100	+ POST /controller/register (with instance metadata, capabilities)
		101	+ → 200: registration accepted, break
		102	+ → non-200: retry with exponential backoff (2, 4, 8, 16, 32s, max 5 retries)
		103	+ → max retries exceeded: os.kill(SIGTERM) — pod restart by k8s
		104	+
		105	+4. Main loop (Application.run()) blocks until stop_event
		106	+ - SIGTERM/SIGINT → cleanup and exit
		107	+ - stdin EOF → cleanup and exit
		108	+ - each daemon tick also checks the HeartbeatManager suicide flag
		109	+
56	```	110	```
57		111
58	`EngineManager` and `HeartbeatManager` are initialized only when the active service registry	112	`EngineManager` and `HeartbeatManager` are initialized only when the active service registry
	@@ -133,11 +187,43 @@ HTTP status failures, TLS errors, connection errors and other exceptions are not
133	`ProcessSupervisor.state()` follows this state model:	187	`ProcessSupervisor.state()` follows this state model:
134		188
135	``` text	189	``` text
136	-start → STARTING	190	+Grace period: 120s hardcoded from engine start
137	- ├─ headless process alive → RUNNING	191	+ (engines need time to load models — don't kill them during warmup)
138	- ├─ /health success → READY	192	+
139	- ├─ process exits → STOPPED	193	+Daemon suicide arbitration (loop paced by heartbeat_interval_seconds,
140	- └─ startup timeout expires while probe fails → UNHEALTHY	194	+threshold 5 ≈ 15s of continuous ABNORMAL with the default 3s interval):
		195	+ - endpoint generation change → counter reset (endpoints were (re)set)
		196	+ - freeze window (engine relaunch / successful death report) → counting
		197	+ suspended; deadline-based freeze, so a lost abort message still expires
		198	+ - has_abnormal_endpoints → counter += 1; else reset + clear report dedup
		199	+ - counter >= 5 → _should_suicide = True
		200	+ → NodeManager._on_daemon_tick() detects the flag → stop_event.set()
		201	+ → main loop exits → shutdown() gracefully stops all modules
		202	+ (Daemon.stop SIGKILLs engine subprocesses — no os._exit(-1))
		203	+ → run() returns exit code -1
		204	+ → Kubernetes restarts the pod → fresh registration
		205	+
		206	+Engine death detection (two signal sources, both report to Controller via
		207	+report_software_fault and freeze suicide ONLY on a successful report —
		208	+dedup by PID / endpoint id):
		209	+ 1. process monitor (5s): engine PID death (EngineDeadError)
		210	+ 2. arbitration: ABNORMAL endpoint that was NORMAL before — covers
		211	+ EngineServer alive but executor dead (vLLM EngineCore crash)
		212	+ Cold-start guard: an endpoint never NORMAL yet is still loading — no report.
		213	+ Report failure (Controller unreachable) → NO freeze: the arbitration keeps
		214	+ counting and the container-restart fallback stays live (freezing on a
		215	+ failed report would leave the pod permanently dead-ended).
		216	+
		217	+Controller-dispatched suicide — POST /node-manager/stop:
		218	+ Daemon.stop() (kill engines) + delayed SIGTERM self → graceful shutdown →
		219	+ exit -1 → k8s restarts the pod. Used for instance teardown and for
		220	+ partial-loss coordination (a surviving NodeManager exits so the whole
		221	+ cross-machine instance restarts together).
		222	+
		223	+HTTP 503 from Controller:
		224	+ → Controller restarted → HeartbeatManager._reregister()
		225	+ → POST /controller/reregister (ReregisterMsg) — single attempt, no backoff;
		226	+ a later heartbeat exception triggers the next retry
141	```	227	```
142		228
143	During `startup_timeout` a failed probe keeps `STARTING`; this prevents slow model loading from	229	During `startup_timeout` a failed probe keeps `STARTING`; this prevents slow model loading from
	@@ -174,10 +260,13 @@ non-headless endpoints. `resume` changes only `PAUSED` records back to `NORMAL`.
174		260
175	### Phase 4: Fault Detection and Recovery	261	### Phase 4: Fault Detection and Recovery
176		262
177	-`Daemon` calls each service's `health_check()` every 5 seconds. If `NativeEngineService` observes a dead	263	+`Daemon` calls each service's `health_check()` every 5 seconds. `NativeEngineService` surfaces dead
178	-native launcher, it removes the record, cleans the process group, and requests one Pod-level	264	+PIDs (`(pid, endpoint_id)`); the Daemon decides the recovery path. Dead engines are reported to the
179	-recovery through `SIGTERM` when `motor_restart_engine` is enabled. `_recovery_requested` prevents	265	+Controller via the software-fault channel; when `enable_engine_relaunch` is set (NodeManager
180	-duplicate recovery signals until a successful new pull resets it.	266	+`fault_tolerance_config`, mirroring the Controller-side switch of the same name) the suicide
		267	+arbitration is frozen for `engine_restart_wait_timeout_sec` so the in-place relaunch can complete.
		268	+Without the switch the freeze is skipped and the pod self-terminates (k8s restarts the container)
		269	+once the abnormal-report threshold is reached.
181		270
182	`HeartbeatManager` counts consecutive successful heartbeat reports containing `ABNORMAL` endpoints.	271	`HeartbeatManager` counts consecutive successful heartbeat reports containing `ABNORMAL` endpoints.
183	After five consecutive abnormal reports it sets the suicide flag. The main application tick sees	272	After five consecutive abnormal reports it sets the suicide flag. The main application tick sees
	@@ -188,11 +277,15 @@ shared long-lived client with bounded retry behavior in `ControllerApiClient`.
188		277
189	### Phase 5: Shutdown	278	### Phase 5: Shutdown
190		279
191	-Shutdown stops modules in reverse initialization order. `Daemon.stop()` asks each service to stop;	280	+``` text
192	-`NativeEngineService` marks records `STOPPING`, sends SIGTERM to each native process group, and waits up to	281	+NodeManager.shutdown() — stop modules in reverse registration order:
193	-the configured grace period for the launcher. It then checks the cached PGID: a group still present after	282	+ HeartbeatManager.stop() → join threads
194	-the launcher exits is force-killed too, so surviving workers cannot retain NPU, port or memory resources.	283	+ RegisterManager.stop() → stop registration thread
195	-Records are removed after cleanup so concurrent status reads cannot report a stopped process as a fresh endpoint.	284	+ NodeManagerAPI.stop() → shutdown FastAPI
		285	+ Daemon.stop() → stop 5s monitor thread, then stop each service
		286	+ in reverse registration order (SIGKILL engine PIDs)
		287	+ (there is no NodeManagerConfig.stop() — the config object has no lifecycle)
		288	+```
196		289
197	### Snapshot Boundary	290	### Snapshot Boundary
198		291

M.agent/skills/motor-dev/references/testing-guide.md+7-0

	@@ -72,6 +72,13 @@ motor/controller/core/bar.py → tests/controller/core/test_bar.py
72		72
73	If no test file exists, create one following `test_<module_name>.py`.	73	If no test file exists, create one following `test_<module_name>.py`.
74		74
		75	+## Test Style Conventions
		76	+
		77	+- 平铺函数，不用测试类：新增测试一律写模块级 `def test_*` 函数，不新建 `class TestXxx`（既有文件已用类的，跟随该文件既有风格）。
		78	+- 不用 `# -- xxx ----` 章节分隔注释与多余空行：函数之间标准 2 空行即可，分隔注释是多行视觉噪音。
		79	+- 不测纯日志输出：日志文案（打印了哪个 banner、分隔符等）是实现细节，不是可观察行为——只断言通过公共 API 可观察的行为。
		80	+- 测试量随功能复杂度：一个「小功能」不需要十几个用例；同类输入用 `@pytest.mark.parametrize` 合并，同函数多个分支优先合并到一个用例里断言（原则 4：测试 diff 明显大于代码 diff 就先砍）。
		81	+
75	## Mock & Fixture Patterns	82	## Mock & Fixture Patterns
76		83
77	- Mocking external services: use `unittest.mock.patch` for HTTP clients and network calls	84	- Mocking external services: use `unittest.mock.patch` for HTTP clients and network calls

Mdocs/zh/design/fault_tolerance/fault_manager.md+2-1

	@@ -53,10 +53,11 @@ Controller 侧:
53	└── Coordinator: 请求异常、熔断、实例隔离与恢复探测	53	└── Coordinator: 请求异常、熔断、实例隔离与恢复探测
54		54
55	NodeManager 侧:	55	NodeManager 侧:
56	- FaultReporter (EngineManager 聚合)	56	+ FaultReporter (Daemon 持有)
57	├── HTTP 轮询 → GET {endpoint.business_port}/fault_tolerance/status (vLLM FT API)	57	├── HTTP 轮询 → GET {endpoint.business_port}/fault_tolerance/status (vLLM FT API)
58	├── 状态去重 → 仅上报 dead/unhealthy 变更	58	├── 状态去重 → 仅上报 dead/unhealthy 变更
59	├── 连续 max_poll_failures 次轮询失败 → 按 dead 上报	59	├── 连续 max_poll_failures 次轮询失败 → 按 dead 上报
		60	+ ├── 引擎重拉期间由 Daemon 暂停/恢复（重拉后清空轮询状态重起启动宽限）
60	└── HTTP POST → Controller /controller/report_software_fault	61	└── HTTP POST → Controller /controller/report_software_fault
61	```	62	```
62		63

Mdocs/zh/design/fault_tolerance/overview.md+3-3

Mdocs/zh/developer_guide/components/node_manager.md+21-24

	@@ -18,17 +18,15 @@ Node Manager 是部署在推理节点上的管理进程，负责连接 Controlle
18	\| `NodeManager` \| `motor/node_manager/node_manager.py` \| `Application` 子类：组装模块并运行 daemon loop，每 tick 检查自杀标志 \|	18	\| `NodeManager` \| `motor/node_manager/node_manager.py` \| `Application` 子类：组装模块并运行 daemon loop，每 tick 检查自杀标志 \|
19	\| `NodeManagerConfig` \| `motor/config/node_manager.py` \| 加载、校验和重载节点配置，推导 endpoint 数量与端口 \|	19	\| `NodeManagerConfig` \| `motor/config/node_manager.py` \| 加载、校验和重载节点配置，推导 endpoint 数量与端口 \|
20	\| `NodeManagerAPI` \| `motor/node_manager/api_server/node_manager_api.py` \| 在后台线程中运行 FastAPI/uvicorn，提供启动、停止和探针接口 \|	20	\| `NodeManagerAPI` \| `motor/node_manager/api_server/node_manager_api.py` \| 在后台线程中运行 FastAPI/uvicorn，提供启动、停止和探针接口 \|
21	-\| `Daemon` \| `motor/node_manager/core/daemon.py` \| 服务编排器：根据配置发现并实例化原生引擎和 KV-store 服务，维护进程监控器 \|	21	+\| `Daemon` \| `motor/node_manager/core/daemon.py` \| 服务编排器：根据配置发现并实例化 Engine 和 KV-store 服务，维护进程监控器与自杀仲裁线程，持有 FaultReporter \|
22	-\| `NativeEngineService` \| `motor/node_manager/core/services/native_engine/service.py` \| 构造不可变 LaunchContext，选择 Native Engine Backend 并管理原生引擎生命周期 \|	22	+\| `EngineService` \| `motor/node_manager/core/services/engine.py` \| Engine 子进程生命周期管理：组装命令、拉起/追踪/停止 `engine_server` 进程、重拉编排（`restart`）、端口就绪等待（`wait_ready`） \|
23	\| `LocalService` \| `motor/node_manager/core/services/memcache/lifecycle.py` \| memcache 后端生命周期管理：配置准备、子进程拉起（通过 `memcache/worker.py`）、健康检查与重启 \|	23	\| `LocalService` \| `motor/node_manager/core/services/memcache/lifecycle.py` \| memcache 后端生命周期管理：配置准备、子进程拉起（通过 `memcache/worker.py`）、健康检查与重启 \|
24	-\| `ProcessSupervisor` \| `motor/node_manager/core/services/native_engine/supervisor.py` \| 创建独立进程组，维护运行态、原生健康探测和完整进程树清理 \|	24	+\| `RegisterManager` \| `motor/node_manager/core/register_manager.py` \| 注册/重注册、校验启动命令、处理 ranktable、快照元数据、持久化引擎重拉参数 \|
25	-\| `VllmBackend` / `SGLangBackend` \| `motor/node_manager/core/services/native_engine/backends/` \| 将统一启动上下文转换为引擎原生命令与 ProbeSpec \|	25	+\| `HeartbeatManager` \| `motor/node_manager/core/heartbeat_manager.py` \| 轮询 endpoint 状态、上报心跳、维护暂停/恢复状态；仅报告状态事实，自杀裁决在 Daemon \|
26	-\| `EngineManager` \| `motor/node_manager/core/engine_manager.py` \| 注册/重注册、校验启动命令、处理 ranktable、快照元数据和故障上报 \|	26	+\| `FaultReporter` \| `motor/node_manager/core/fault_reporter.py` \| 轮询引擎 FT 状态接口并上报软件故障给 Controller；由 Daemon 持有，重拉期间暂停/恢复 \|
27	-\| `HeartbeatManager` \| `motor/node_manager/core/heartbeat_manager.py` \| 轮询 endpoint 状态、上报心跳、维护暂停/恢复状态并触发异常自杀 \|
28	-\| `FaultReporter` \| `motor/node_manager/core/fault_reporter.py` \| 轮询引擎 FT 状态接口并上报软件故障给 Controller \|
29	\| `ControllerApiClient` \| `motor/node_manager/api_client/controller_api_client.py` \| 调用 Controller 的注册、重注册、心跳和故障上报接口 \|	27	\| `ControllerApiClient` \| `motor/node_manager/api_client/controller_api_client.py` \| 调用 Controller 的注册、重注册、心跳和故障上报接口 \|
30		28
31	-`Daemon`、`EngineManager` 和 `HeartbeatManager` 均为线程安全单例。HTTP 路由和后台线程通过这些单例共享实例、endpoint 和进程状态。`Application` 和 `NodeManager` 不是单例，由 `main.py` 显式创建。	29	+`Daemon`、`RegisterManager` 和 `HeartbeatManager` 均为线程安全单例。HTTP 路由和后台线程通过这些单例共享实例、endpoint 和进程状态。`Application` 和 `NodeManager` 不是单例，由 `main.py` 显式创建。
32		30
33	## 生命周期	31	## 生命周期
34		32
	@@ -39,13 +37,13 @@ Node Manager 是部署在推理节点上的管理进程，负责连接 Controlle
39	1. 模块级 `set_process_title("NodeManager")` 设置进程名。	37	1. 模块级 `set_process_title("NodeManager")` 设置进程名。
40	2. `main()` 加载 `NodeManagerConfig`，配置日志，执行端口分配。	38	2. `main()` 加载 `NodeManagerConfig`，配置日志，执行端口分配。
41	3. 创建 `NodeManager(config)` 并调用 `run()`，内部执行：	39	3. 创建 `NodeManager(config)` 并调用 `run()`，内部执行：
42	- a. `init_modules()` — 根据 `Daemon.has_engine` 动态注册模块：`Daemon`、`NodeManagerAPI`，以及有 Engine 时才注册的 `EngineManager` 和 `HeartbeatManager`。	40	+ a. `init_modules()` — 根据 `Daemon.has_engine` 动态注册模块：`Daemon`、`NodeManagerAPI`，以及有 Engine 时才注册的 `RegisterManager` 和 `HeartbeatManager`。
43	b. `_start_config_watcher()` — 非快照模式下启动配置文件 watcher；快照模式跳过。	41	b. `_start_config_watcher()` — 非快照模式下启动配置文件 watcher；快照模式跳过。
44	c. `setup_signal_handlers()` — 注册 SIGINT / SIGTERM。	42	c. `setup_signal_handlers()` — 注册 SIGINT / SIGTERM。
45	d. `_daemon_loop()` — select-based 主循环，每 `daemon_loop_interval` 秒检查自杀标志和 stdin 输入。	43	d. `_daemon_loop()` — select-based 主循环，每 `daemon_loop_interval` 秒检查自杀标志和 stdin 输入。
46	-4. 各模块的初始化行为不变：`NodeManagerAPI.__init__` 在后台线程中启动 FastAPI，`EngineManager.__init__` 启动注册线程，`Daemon.__init__` 启动进程监控线程。	44	+4. 各模块的初始化行为不变：`NodeManagerAPI.__init__` 在后台线程中启动 FastAPI，`RegisterManager.__init__` 启动注册线程，`Daemon.__init__` 启动进程监控线程。
47		45
48	-首次注册会持续重试直至成功，重试间隔为 2、4、8、16、32 秒（上限 32 秒）。注册失败不会向当前进程发送 `SIGTERM`。	46	+首次注册最多尝试 5 次，重试间隔为 2、4、8、16 秒。连续失败后，`RegisterManager` 向当前进程发送 `SIGTERM`。
49		47
50	### 启动实例	48	### 启动实例
51		49
	@@ -55,20 +53,17 @@ Controller 调用 `POST /node-manager/start` 后，处理流程为：
55	2. 校验 `job_name`、endpoint 数量以及每个 endpoint 的 IP 是否与本节点配置一致。	53	2. 校验 `job_name`、endpoint 数量以及每个 endpoint 的 IP 是否与本节点配置一致。
56	3. 保存 `instance_id`、endpoints、`node_rank` 和 D2D peer 信息；如配置了 `RANKTABLE_PATH`，将实例 ranktable 写入该文件。	54	3. 保存 `instance_id`、endpoints、`node_rank` 和 D2D peer 信息；如配置了 `RANKTABLE_PATH`，将实例 ranktable 写入该文件。
57	4. 准备快照运行目录和元数据。	55	4. 准备快照运行目录和元数据。
58	-5. `Daemon.pull_engine()` 为每个 endpoint 直接拉起一个原生 vLLM 或 SGLang 进程组。	56	+5. `Daemon.pull_engine()` 为每个 endpoint 拉起一个 `engine_server` 子进程，并启动 `Daemon` 持有的 `FaultReporter`（仅在故障容忍功能开启时生效）。
59	6. 更新 `HeartbeatManager` 中的 endpoint，并启动状态轮询和心跳线程。	57	6. 更新 `HeartbeatManager` 中的 endpoint，并启动状态轮询和心跳线程。
60	-7. 启动 `EngineManager` 中的 `FaultReporter`（仅在故障容忍功能开启时生效）。
61		58
62	从宿主机侧快照恢复时，第 5 步不会再次拉起引擎，而是更新恢复元数据、endpoint 和恢复状态。	59	从宿主机侧快照恢复时，第 5 步不会再次拉起引擎，而是更新恢复元数据、endpoint 和恢复状态。
63		60
64	### 停止与重调度	61	### 停止与重调度
65		62
66	-- 收到 `SIGINT`、`SIGTERM` 或标准输入命令 `stop` 时，`Application._handle_signal()` 设置 `stop_event`；daemon loop 退出后停止配置 watcher，并按初始化的逆序停止模块。	63	+- 收到 `SIGINT`、`SIGTERM` 或标准输入命令 `stop` 时，`Application._handle_signal()` 设置 `stop_event`，daemon loop 退出后执行 `shutdown()`：按注册逆序调用每个模块的 `stop()`，然后停止配置 watcher。
67	-- `Daemon.stop()` 通过 `ProcessSupervisor` 向所有原生引擎进程组发送 `SIGTERM`；宽限期后仍未退出时发送 `SIGKILL` 清理完整进程树。	64	+- `Daemon.stop()` 遍历所有 service 调用 `stop()`：`EngineService.stop()` 对记录的 Engine Server PID 发送 `SIGKILL`；`LocalService.stop()` 对 memcache worker 子进程发送 `SIGKILL`。
68	-- `Daemon.stop()` 同时遍历其他已启用 service；例如 `LocalService.stop()` 会停止 memcache worker 子进程。	65	+- 自杀裁决由 `Daemon` 独立 3s 仲裁线程执行：任一 endpoint 连续 5 轮观察保持 `ABNORMAL`（约 15s）时设置自杀标志（引擎重拉/死亡上报的冻结窗口内暂停计数）。daemon loop 每 tick 检查该标志，触发后 `stop_event.set()` 并返回 `-1`，用于触发重调度。
69	-- 任一原生引擎进程异常退出时只触发一次 Pod 级恢复，不在 Pod 内重启单个 rank。	66	+- `exit_code` 默认返回 `-1`，与旧行为一致（-1 表示 rescheduling）。
70	-- 任一 endpoint 连续 5 个心跳周期保持 `ABNORMAL` 时，`HeartbeatManager` 设置自杀标志。主线程执行清理后返回 `-1`，用于触发重调度。
71	-- 当前 `main()` 正常退出路径同样返回 `-1`；源码注释约定 `-1` 表示 rescheduling、`0` 表示 restart。
72		67
73	## Node Manager HTTP API	68	## Node Manager HTTP API
74		69
	@@ -76,11 +71,12 @@ Node Manager API 默认监听 `api_config.pod_ip:api_config.node_manager_port`
76		71
77	\| 方法 \| 路径 \| 响应 \| 说明 \|	72	\| 方法 \| 路径 \| 响应 \| 说明 \|
78	\|------\|------\|----------\|------\|	73	\|------\|------\|----------\|------\|
79	-\| `POST` \| `/node-manager/start` \| `200 {}` \| 校验启动命令并拉起原生引擎；快照恢复时执行恢复准备 \|	74	+\| `POST` \| `/node-manager/start` \| `200 {}` \| 校验启动命令并拉起 Engine Server；快照恢复时执行恢复准备 \|
80	-\| `POST` \| `/node-manager/stop` \| `200 {"message": "All engine processes stopped successfully."}` \| 停止当前 Node Manager 监管的全部原生引擎进程组 \|	75	+\| `POST` \| `/node-manager/stop` \| `200 {"message": "All engine processes stopped successfully."}` \| 停止全部 Engine Server 进程后延时 SIGTERM 自身（退出码 `-1` → k8s 重启 Pod），即 Controller 下发的「自杀」指令，用于实例拆除与跨机部分失联协同 \|
81	-\| `POST` \| `/node-manager/pause` \| `200 {"status":"ok", ...}` \| 将全部 endpoint 标记为 `PAUSED`，并返回非 headless 原生引擎 metrics URL \|	76	+\| `POST` \| `/node-manager/engine-restart` \| `200 {"message": ...}` \| Controller 驱动的容器内引擎重拉：body `{"action": "restart"\\|"abort", "instance_id"?}`。`restart` 整体委托 `Daemon.restart_engine`（Daemon 解析启动参数、冻结自杀仲裁、暂停/恢复 FaultReporter、杀掉并重拉全部引擎，KV store 不动）；`abort` = 解冻自杀仲裁（重拉失败回退容器重启）。并发 409、快照恢复中 409、无启动记录 400、重拉失败 500 且解冻 \|
		77	+\| `POST` \| `/node-manager/pause` \| `200 {"status":"ok", ...}` \| 将全部 endpoint 标记为 `PAUSED`，并返回 Engine Server 管理地址 \|
82	\| `POST` \| `/node-manager/resume` \| `200 {"status":"ok", ...}` \| 仅将 `PAUSED` endpoint 恢复为 `NORMAL` \|	78	\| `POST` \| `/node-manager/resume` \| `200 {"status":"ok", ...}` \| 仅将 `PAUSED` endpoint 恢复为 `NORMAL` \|
83	-\| `GET` \| `/node-manager/status` \| `200 {"status": true/false}` \| 返回全部 endpoint 是否为 `NORMAL`；无 endpoint 时为 `false` \|	79	+\| `GET` \| `/node-manager/status` \| `200 {"status": true/false}` \| 返回全部 endpoint 是否为 `NORMAL`；`relaxed=true`（引擎重拉轮询）时无 `ABNORMAL` 即 `true`；无 endpoint 时为 `false` \|
84	\| `GET` \| `/readiness` \| `200` 或 `503` \| Kubernetes Readiness Probe 接口。实例节点 Pod 默认不配置该探针；仅在容器快照默认应用场景下配置，用于判断执行容器 checkpoint 前的稳态点。未到达稳态点时返回 `503`，到达后返回 `200` \|	80	\| `GET` \| `/readiness` \| `200` 或 `503` \| Kubernetes Readiness Probe 接口。实例节点 Pod 默认不配置该探针；仅在容器快照默认应用场景下配置，用于判断执行容器 checkpoint 前的稳态点。未到达稳态点时返回 `503`，到达后返回 `200` \|
85		81
86	`/node-manager/pause` 用于 PreStop 优雅下线：暂停状态会使 readiness 失败，并通过心跳通知 Controller；原生健康轮询不会覆盖手动设置的 `PAUSED`。响应中的 `engine_metrics_targets` 使用原生业务端口，并排除 headless 成员。如果 PreStop 被取消，可调用 `/node-manager/resume` 恢复调度。	82	`/node-manager/pause` 用于 PreStop 优雅下线：暂停状态会使 readiness 失败，并通过心跳通知 Controller；原生健康轮询不会覆盖手动设置的 `PAUSED`。响应中的 `engine_metrics_targets` 使用原生业务端口，并排除 headless 成员。如果 PreStop 被取消，可调用 `/node-manager/resume` 恢复调度。
	@@ -202,7 +198,8 @@ Node Manager 从 `engine_config.nnodes` 推导每节点 `local_world_size`。当
202	1. `_refresh_check_interval()` — 从配置刷新 daemon loop 间隔。	198	1. `_refresh_check_interval()` — 从配置刷新 daemon loop 间隔。
203	2. 遍历所有模块调用 `update_config()`：	199	2. 遍历所有模块调用 `update_config()`：
204	- `HeartbeatManager` 动态更新 `heartbeat_interval_seconds`。	200	- `HeartbeatManager` 动态更新 `heartbeat_interval_seconds`。
205	- - `EngineManager` 更新配置，并根据 `enable_fault_tolerance`、endpoint 的变化启停或重建 `FaultReporter`。	201	+ - `Daemon` 更新配置，并根据 `enable_fault_tolerance`、endpoint 的变化启停或重建其持有的 `FaultReporter`。
		202	+ - `RegisterManager` 更新配置。
206	3. 打印更新后的配置摘要 `log_configuration_summary()`。	203	3. 打印更新后的配置摘要 `log_configuration_summary()`。
207	4. API 监听地址、监听端口、TLS 和 `Daemon` 已缓存的设备参数不会热重启，修改后需要重启 Node Manager。	204	4. API 监听地址、监听端口、TLS 和 `Daemon` 已缓存的设备参数不会热重启，修改后需要重启 Node Manager。
208		205

Mexamples/features/config_sample.json+10-2

	@@ -100,7 +100,12 @@
100	"k8s_cert_path": "",	100	"k8s_cert_path": "",
101	"enable_scale_p2d": true,	101	"enable_scale_p2d": true,
102	"enable_token_reinference": true,	102	"enable_token_reinference": true,
103	- "scale_p2d_d_instance_reinit_wait_timeout": 60	103	+ "scale_p2d_d_instance_reinit_wait_timeout": 60,
		104	+ "enable_engine_relaunch": true,
		105	+ "engine_relaunch_complete_timeout_sec": 600,
		106	+ "engine_relaunch_poll_interval_sec": 5.0,
		107	+ "engine_relaunch_dispatch_retries": 3,
		108	+ "engine_relaunch_nm_unreachable_threshold": 3
104	},	109	},
105	"observability_config": {	110	"observability_config": {
106	"observability_enable": false,	111	"observability_enable": false,
	@@ -360,7 +365,10 @@
360	"enable_fault_tolerance": false,	365	"enable_fault_tolerance": false,
361	"poll_interval_sec": 5.0,	366	"poll_interval_sec": 5.0,
362	"poll_timeout_sec": 5.0,	367	"poll_timeout_sec": 5.0,
363	- "max_poll_failures": 3	368	+ "max_poll_failures": 3,
		369	+ "enable_engine_relaunch": true,
		370	+ "engine_restart_wait_timeout_sec": 180.0,
		371	+ "engine_restart_freeze_sec": 720.0
364	},	372	},
365	"port_allocator_config": {	373	"port_allocator_config": {
366	"enable": true,	374	"enable": true,

Mmotor/common/app/application.py+8-1

	@@ -15,6 +15,8 @@ handling, and a select-based daemon loop. Subclasses implement
15	:meth:`init_modules` and optionally override :meth:`run`.	15	:meth:`init_modules` and optionally override :meth:`run`.
16	"""	16	"""
17		17
		18	+import asyncio
		19	+import inspect
18	import select	20	import select
19	import signal	21	import signal
20	import sys	22	import sys
	@@ -95,7 +97,12 @@ class Application(ABC):
95	for name, module in reversed(list(self.modules.items())):	97	for name, module in reversed(list(self.modules.items())):
96	if hasattr(module, "stop"):	98	if hasattr(module, "stop"):
97	try:	99	try:
98	- module.stop()	100	+ result = module.stop()
		101	+ # Modules may declare async stop() (e.g. the FastAPI
		102	+ # server); the daemon main loop is synchronous, so run
		103	+ # the coroutine to completion instead of dropping it.
		104	+ if inspect.iscoroutine(result):
		105	+ asyncio.run(result)
		程哥哥吖12 天前问题：asyncio.run() 会创建并关闭新事件循环，且不能嵌套调用，也不能在已有事件循环的线程中调用。若某个 module 的 stop() 是 async，且当前线程已有事件循环（例如 Coordinator的某些组件、测试事件循环、或被 async 上下文调用时），会抛出 RuntimeError。更严重的是，每个 async stop 都独立 asyncio.run，无法共享 loop。建议：改为 loop = asyncio.new_event_loop(); loop.run_until_complete(coro); loop.close()，或一次性收集所有 async stop 后统一 asyncio.run(asyncio.gather(*coros))。 jason lyu 12 天前评论： 12 天前评论：
99	except Exception:	106	except Exception:
100	logger.exception("Failed to stop module %r", name)	107	logger.exception("Failed to stop module %r", name)
101	self.modules.clear()	108	self.modules.clear()

Mmotor/common/resources/instance.py+16-0

	@@ -359,6 +359,22 @@ class Instance(BaseModel):
359	return False	359	return False
360	return has_routable_endpoint	360	return has_routable_endpoint
361		361
		362	+ def is_all_endpoints_heartbeat_fresh(self, timeout: float = DEFAULT_ACTIVE_HEARTBEAT_TIMEOUT) -> bool:
		363	+ """True when every endpoint's last heartbeat is within ``timeout`` seconds.
		364	+
		365	+ Complements ``is_all_endpoints_ready``: the status field alone goes
		366	+ stale when a NodeManager dies (nobody updates its endpoints), so an
		367	+ instance must not flip ACTIVE on a lone surviving NodeManager's
		368	+ heartbeat while another NodeManager's heartbeats have timed out.
		369	+ """
		370	+ now = time.time()
		371	+ with self._lock:
		372	+ for pod_endpoints in self.endpoints.values():
		373	+ for endpoint in pod_endpoints.values():
		374	+ if not endpoint.is_alive(now, timeout):
		375	+ return False
		376	+ return True
		377	+
362	def is_have_one_endpoint_abnormal(self) -> bool:	378	def is_have_one_endpoint_abnormal(self) -> bool:
363	abnormal_endpoints: dict[str, list[int]] = {} # pod_ip -> [endpoint_id]	379	abnormal_endpoints: dict[str, list[int]] = {} # pod_ip -> [endpoint_id]
364	with self._lock:	380	with self._lock:

Mmotor/common/utils/env.py+0-4

	@@ -98,10 +98,6 @@ class _Environment:
98		98
99	# --- Daemon behaviour ---	99	# --- Daemon behaviour ---
100		100
101	- @property
102	- def motor_restart_engine(self):
103	- return os.getenv("MOTOR_RESTART_ENGINE", "0") == "1"
104	-
105	@property	101	@property
106	def motor_restart_local_service(self):	102	def motor_restart_local_service(self):
107	return os.getenv("MOTOR_RESTART_LOCAL_SERVICE", "1") == "1"	103	return os.getenv("MOTOR_RESTART_LOCAL_SERVICE", "1") == "1"

Mmotor/config/controller.py+27-0

	@@ -128,6 +128,20 @@ class FaultToleranceConfig:
128	enable_token_reinference: bool = True # Enable/disable token reinference strategy	128	enable_token_reinference: bool = True # Enable/disable token reinference strategy
129	scale_p2d_d_instance_reinit_wait_timeout: int = 60 # seconds to wait for D instance re-init before ScaleP2D	129	scale_p2d_d_instance_reinit_wait_timeout: int = 60 # seconds to wait for D instance re-init before ScaleP2D
130		130
		131	+ # Fallback engine relaunch (restart engines in place, then containers):
		132	+ # when an engine is reported DEAD or a fast-recovery strategy fails, the
		133	+ # strategy center escalates to EngineRelaunchStrategy. Disabling restores
		134	+ # the legacy behavior (heartbeat suicide -> k8s pod restart only).
		135	+ enable_engine_relaunch: bool = True
		136	+ #: Total budget for relaunching engines (dispatch + model loading).
		137	+ engine_relaunch_complete_timeout_sec: int = 600
		138	+ #: Polling interval while waiting for the engines to come back.
		139	+ engine_relaunch_poll_interval_sec: float = 5.0
		140	+ #: Dispatch retries (with 2s backoff) per NodeManager.
		141	+ engine_relaunch_dispatch_retries: int = 3
		142	+ #: Consecutive poll failures before a NodeManager counts as unreachable.
		143	+ engine_relaunch_nm_unreachable_threshold: int = 3
		144	+
131		145
132	@dataclass	146	@dataclass
133	class ControllerConfig:	147	class ControllerConfig:
	@@ -304,6 +318,19 @@ class ControllerConfig:
304	if not (1 <= self.fault_tolerance_config.scale_p2d_d_instance_reinit_wait_timeout <= 600):	318	if not (1 <= self.fault_tolerance_config.scale_p2d_d_instance_reinit_wait_timeout <= 600):
305	errors.append("scale_p2d_d_instance_reinit_wait_timeout must be in range 1-600")	319	errors.append("scale_p2d_d_instance_reinit_wait_timeout must be in range 1-600")
306		320
		321	+ ft_config = self.fault_tolerance_config
		322	+ if not (60 <= ft_config.engine_relaunch_complete_timeout_sec <= 3600):
		323	+ errors.append("engine_relaunch_complete_timeout_sec must be in range 60-3600")
		324	+
		325	+ if not (1 <= ft_config.engine_relaunch_poll_interval_sec <= 60):
		326	+ errors.append("engine_relaunch_poll_interval_sec must be in range 1-60")
		327	+
		328	+ if not (0 <= ft_config.engine_relaunch_dispatch_retries <= 10):
		329	+ errors.append("engine_relaunch_dispatch_retries must be in range 0-10")
		330	+
		331	+ if not (1 <= ft_config.engine_relaunch_nm_unreachable_threshold <= 10):
		332	+ errors.append("engine_relaunch_nm_unreachable_threshold must be in range 1-10")
		333	+
307	# Validate standby configuration	334	# Validate standby configuration
308	if self.standby_config.master_standby_check_interval <= 0:	335	if self.standby_config.master_standby_check_interval <= 0:
309	errors.append("master_standby_check_interval must be greater than 0")	336	errors.append("master_standby_check_interval must be greater than 0")

Mmotor/config/node_manager.py+27-0

	@@ -296,6 +296,25 @@ class NodeManagerFaultToleranceConfig:
296	#: HTTP timeout for a single status poll.	296	#: HTTP timeout for a single status poll.
297	max_poll_failures: int = 3	297	max_poll_failures: int = 3
298	#: Consecutive poll failures before the engine is reported as dead.	298	#: Consecutive poll failures before the engine is reported as dead.
		299	+ #: In-place engine relaunch (container keeps running). Mirrors the
		300	+ #: Controller-side switch of the same name — the NodeManager holds its
		301	+ #: own copy because the two run in separate processes with separate
		302	+ #: configs. Disabling it restores the legacy behavior: a dead engine is
		303	+ #: reported but the suicide arbitration is not frozen, so the pod
		304	+ #: self-terminates (k8s restarts the container) instead.
		305	+ enable_engine_relaunch: bool = True
		306	+ #: Suicide-freeze window after a successful death report while waiting
		307	+ #: for the Controller to dispatch the relaunch. Must cover the worst-case
		308	+ #: dispatch latency: death report (~15s) + strategy-center cycle
		309	+ #: (strategy_center_check_interval, default 10s) + serial probe/dispatch
		310	+ #: per NodeManager (retries + backoff). A false alarm unfreezes when the
		311	+ #: deadline expires.
		312	+ engine_restart_wait_timeout_sec: float = 180.0
		313	+ #: Suicide-freeze window while engines are being re-pulled. Must exceed
		314	+ #: the Controller relaunch completion timeout (600s) plus margin; the
		315	+ #: deadline (not a boolean) keeps the container-restart fallback alive
		316	+ #: even when the abort message is lost.
		317	+ engine_restart_freeze_sec: float = 720.0
299		318
300		319
301	@dataclass	320	@dataclass
	@@ -804,6 +823,14 @@ class NodeManagerConfig:
804	if self.logging_config.log_max_line_length <= 0:	823	if self.logging_config.log_max_line_length <= 0:
805	errors.append("log_max_line_length must be greater than 0")	824	errors.append("log_max_line_length must be greater than 0")
806		825
		826	+ # Validate fault tolerance configuration
		827	+ ft_config = self.fault_tolerance_config
		828	+ if not (60 <= ft_config.engine_restart_wait_timeout_sec <= 3600):
		829	+ errors.append("engine_restart_wait_timeout_sec must be in range 60-3600")
		830	+
		831	+ if not (120 <= ft_config.engine_restart_freeze_sec <= 7200):
		832	+ errors.append("engine_restart_freeze_sec must be in range 120-7200")
		833	+
807	raise_if_config_errors(errors)	834	raise_if_config_errors(errors)
808		835
809	def reload(self) -> bool:	836	def reload(self) -> bool:

Mmotor/controller/api_client/node_manager_api_client.py+61-5

	@@ -1,4 +1,3 @@
1	-# -- coding: utf-8 --
2	# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.	1	# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
3	# MindIE is licensed under Mulan PSL v2.	2	# MindIE is licensed under Mulan PSL v2.
4	# You can use this software according to the terms and conditions of the Mulan PSL v2.	3	# You can use this software according to the terms and conditions of the Mulan PSL v2.
	@@ -8,8 +7,11 @@
8	# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,	7	# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
9	# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.	8	# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
10	# See the Mulan PSL v2 for more details.	9	# See the Mulan PSL v2 for more details.
		10	+
11	from typing import Any	11	from typing import Any
12		12
		13	+import requests
		14	+
13	from motor.common.resources import NodeManagerInfo, StartCmdMsg	15	from motor.common.resources import NodeManagerInfo, StartCmdMsg
14	from motor.common.http.http_client import SafeHTTPSClient	16	from motor.common.http.http_client import SafeHTTPSClient
15	from motor.common.logger import get_logger	17	from motor.common.logger import get_logger
	@@ -25,6 +27,7 @@ class NodeManagerApiClient:
25	@staticmethod	27	@staticmethod
26	def send_start_command(node_mgr: NodeManagerInfo, start_cmd_msg: StartCmdMsg) -> bool:	28	def send_start_command(node_mgr: NodeManagerInfo, start_cmd_msg: StartCmdMsg) -> bool:
27	is_succeed = True	29	is_succeed = True
		30	+ client = None
28	try:	31	try:
29	# For `superpod_id` we need to use `exclude_none` to avoid error,	32	# For `superpod_id` we need to use `exclude_none` to avoid error,
30	# when we use atlas A2 server which doesn't have superpod_id.	33	# when we use atlas A2 server which doesn't have superpod_id.
	@@ -48,7 +51,8 @@ class NodeManagerApiClient:
48	e,	51	e,
49	)	52	)
50	finally:	53	finally:
51	- client.close()	54	+ if client is not None:
		55	+ client.close()
52		56
53	return is_succeed	57	return is_succeed
54		58
	@@ -56,6 +60,7 @@ class NodeManagerApiClient:
56	def stop(node_mgr: NodeManagerInfo) -> bool:	60	def stop(node_mgr: NodeManagerInfo) -> bool:
57	is_succeed = True	61	is_succeed = True
58	addr = format_address(node_mgr.pod_ip, node_mgr.port)	62	addr = format_address(node_mgr.pod_ip, node_mgr.port)
		63	+ client = None
59	try:	64	try:
60	client_args = NodeManagerApiClient._generate_client_args(node_mgr)	65	client_args = NodeManagerApiClient._generate_client_args(node_mgr)
61	client = SafeHTTPSClient(**client_args)	66	client = SafeHTTPSClient(**client_args)
	@@ -65,15 +70,66 @@ class NodeManagerApiClient:
65	is_succeed = False	70	is_succeed = False
66	logger.error("Error sending stop command to node manager %s: %s", addr, e)	71	logger.error("Error sending stop command to node manager %s: %s", addr, e)
67	finally:	72	finally:
68	- client.close()	73	+ if client is not None:
		74	+ client.close()
		75	+
		76	+ return is_succeed
		77	+
		78	+ @staticmethod
		79	+ def restart_engine(node_mgr: NodeManagerInfo, action: str = "restart", instance_id: int \| None = None) -> bool:
		80	+ """Dispatch the engine relaunch (or its abort) to a NodeManager.
		81	+
		82	+ ``action="restart"``: kill and re-pull all engines without restarting
		83	+ the container. ``action="abort"``: unfreeze the NodeManager's suicide
		84	+ counter so the heartbeat mechanism restarts the container (fallback
		85	+ when relaunch failed). Returns False when the NodeManager is
		86	+ unreachable or rejects the request.
		87	+
		88	+ A 409 (relaunch already in progress on that NodeManager) counts as
		89	+ success: the work is being done there — treating it as a dispatch
		90	+ failure would retry and then wrongly escalate to container restart
		91	+ while the engines are actually being relaunched.
		92	+ """
		93	+ is_succeed = True
		94	+ addr = format_address(node_mgr.pod_ip, node_mgr.port)
		95	+ data = {"action": action}
		96	+ if instance_id is not None:
		97	+ data["instance_id"] = instance_id
		98	+ client = None
		99	+ try:
		程哥哥吖12 天前如果 _generate_client_args 或 SafeHTTPSClient 构造失败，client 未定义，finally 中 client.close() 会抛出 UnboundLocalError，掩盖原始异常。建议：将 client = SafeHTTPSClient(...) 提前到 try 之前，或初始化 client = None 并在 finally 中判断。 jason lyu 12 天前评论： 12 天前评论：
		100	+ client_args = NodeManagerApiClient._generate_client_args(node_mgr)
		101	+ client = SafeHTTPSClient(**client_args)
		102	+ client.post("/node-manager/engine-restart", data=data)
		103	+ logger.info("Engine restart command (%s) sent to node manager %s", action, addr)
		104	+ except requests.exceptions.HTTPError as e:
		105	+ if e.response is not None and e.response.status_code == 409:
		106	+ logger.info(
		107	+ "Engine relaunch already in progress on node manager %s (409), treating as dispatched", addr
		108	+ )
		109	+ else:
		110	+ is_succeed = False
		111	+ logger.error("Error sending engine restart command (%s) to node manager %s: %s", action, addr, e)
		112	+ except Exception as e:
		113	+ is_succeed = False
		114	+ logger.error("Error sending engine restart command (%s) to node manager %s: %s", action, addr, e)
		115	+ finally:
		116	+ if client is not None:
		117	+ client.close()
69		118
70	return is_succeed	119	return is_succeed
71		120
72	@classmethod	121	@classmethod
73	- def query_status(cls, node_mgr: NodeManagerInfo) -> dict[str, Any]:	122	+ def query_status(cls, node_mgr: NodeManagerInfo, relaxed: bool = False) -> dict[str, Any]:
		123	+ """Query the NodeManager's endpoint readiness.
		124	+
		125	+ ``relaxed=True`` (engine-relaunch flow): True when no endpoint is
		126	+ ABNORMAL — a freshly relaunched engine is INITIAL (model loading) and
		127	+ counts as recovering; the strict mode requires all NORMAL.
		128	+ """
74	client_args = NodeManagerApiClient._generate_client_args(node_mgr)	129	client_args = NodeManagerApiClient._generate_client_args(node_mgr)
75	client = SafeHTTPSClient(**client_args)	130	client = SafeHTTPSClient(**client_args)
76	- response = client.get("/node-manager/status")	131	+ params = {"relaxed": "true"} if relaxed else None
		132	+ response = client.get("/node-manager/status", params=params)
77	return response	133	return response
78		134
79	@classmethod	135	@classmethod

Mmotor/controller/core/instance_manager.py+66-7

	@@ -127,6 +127,8 @@ class InstanceManager(ThreadSafeSingleton):
127	}	127	}
128		128
129	self.instances_management_thread = None	129	self.instances_management_thread = None
		130	+ # instance ids for which the partial-loss shutdown was already dispatched
		131	+ self._partial_loss_shutdown_sent: set[int] = set()
130		132
131	self._initialized = True	133	self._initialized = True
132	logger.info("InstanceManager initialized.")	134	logger.info("InstanceManager initialized.")
	@@ -638,6 +640,9 @@ class InstanceManager(ThreadSafeSingleton):
638	def _handle_active(self, from_state: InsStatus, condition_event: InsConditionEvent, instance: Instance) -> None:	640	def _handle_active(self, from_state: InsStatus, condition_event: InsConditionEvent, instance: Instance) -> None:
639	if from_state == InsStatus.ACTIVE:	641	if from_state == InsStatus.ACTIVE:
640	return	642	return
		643	+ # The instance recovered (or a new one with this id was assembled) —
		644	+ # allow the shutdown dispatch again for a future loss episode.
		645	+ self._partial_loss_shutdown_sent.discard(instance.id)
641	if condition_event in (InsConditionEvent.INSTANCE_NORMAL, InsConditionEvent.INSTANCE_RESUMED):	646	if condition_event in (InsConditionEvent.INSTANCE_NORMAL, InsConditionEvent.INSTANCE_RESUMED):
642	instance.update_instance_status(InsStatus.ACTIVE)	647	instance.update_instance_status(InsStatus.ACTIVE)
643	if condition_event == InsConditionEvent.INSTANCE_RESUMED:	648	if condition_event == InsConditionEvent.INSTANCE_RESUMED:
	@@ -669,11 +674,13 @@ class InstanceManager(ThreadSafeSingleton):
669	# After failure, node_manager easily reports heartbeats to the wrong pod, leading to	674	# After failure, node_manager easily reports heartbeats to the wrong pod, leading to
670	# heartbeat timeout and instance isolation. Therefore, we need the controller to	675	# heartbeat timeout and instance isolation. Therefore, we need the controller to
671	# directly query once using ip+port when heartbeat times out to avoid such situations.]	676	# directly query once using ip+port when heartbeat times out to avoid such situations.]
672	- if self._check_node_managers_status(instance):	677	+ has_abnormal, reachable_nms = self._check_node_managers_status(instance)
		程哥哥吖12 天前问题：_check_node_managers_status 遇到第一个 abnormal 的 NodeManager 就立即 return True, reachable_nms，此时 reachable_nms 只包含之前检查过的 NodeManager。如果异常 NodeManager 排在前面，返回的 reachable 列表可能为空或不全，导致 _dispatch_partial_loss_shutdown 漏发 shutdown。建议：先遍历全部 NodeManager，收集所有可达的，再统一返回 (has_abnormal, reachable_nms)。 jason lyu 12 天前评论： 12 天前评论：
		678	+ if has_abnormal:
673	instance.update_instance_status(InsStatus.INACTIVE)	679	instance.update_instance_status(InsStatus.INACTIVE)
674	self.notify(instance, ObserverEvent.INSTANCE_SEPARATED)	680	self.notify(instance, ObserverEvent.INSTANCE_SEPARATED)
675	self._report_inst_alarm(instance)	681	self._report_inst_alarm(instance)
676	self._report_coordinator_alarm(instance)	682	self._report_coordinator_alarm(instance)
		683	+ self._dispatch_partial_loss_shutdown(instance, reachable_nms)
677	else:	684	else:
678	# If node managers are all normal, do not set to INACTIVE	685	# If node managers are all normal, do not set to INACTIVE
679	# and we need to refresh the heartbeat to avoid immediate timeout	686	# and we need to refresh the heartbeat to avoid immediate timeout
	@@ -689,6 +696,37 @@ class InstanceManager(ThreadSafeSingleton):
689	self.notify(instance, ObserverEvent.INSTANCE_PAUSED)	696	self.notify(instance, ObserverEvent.INSTANCE_PAUSED)
690	self._report_inst_alarm(instance)	697	self._report_inst_alarm(instance)
691		698
		699	+ def _dispatch_partial_loss_shutdown(self, instance: Instance, reachable_nms: list) -> None:
		700	+ """Tell the surviving NodeManagers to exit after a partial instance loss.
		701	+
		702	+ One NodeManager is gone (heartbeats timed out and the probe confirmed
		703	+ it), so the surviving half must restart together with it — a
		704	+ cross-machine instance cannot run split. ``/node-manager/stop`` is the
		705	+ existing suicide instruction (engines stopped, then the NodeManager
		706	+ exits and k8s restarts the pod). Dispatched once per loss episode; the
		707	+ marker clears when the instance recovers to ACTIVE or is deleted (a
		708	+ re-assembled instance gets a fresh id).
		709	+ """
		710	+ if instance.id in self._partial_loss_shutdown_sent:
		711	+ return
		712	+ self._partial_loss_shutdown_sent.add(instance.id)
		713	+ logger.warning(
		714	+ "Instance %d partially lost: dispatching stop to %d surviving node manager(s)",
		715	+ instance.id,
		716	+ len(reachable_nms),
		717	+ )
		718	+ for node_mgr in reachable_nms:
		719	+ try:
		720	+ NodeManagerApiClient.stop(node_mgr)
		721	+ logger.error(
		722	+ "Stop dispatched to node manager %s:%s (instance %d partially lost)",
		723	+ node_mgr.pod_ip,
		724	+ node_mgr.port,
		725	+ instance.id,
		726	+ )
		727	+ except Exception as e:
		728	+ logger.error("Failed to dispatch stop to node manager %s: %s", node_mgr.pod_ip, e)
		729	+
692	def _report_inst_alarm(self, instance: Instance, is_cleared: bool = False) -> None:	730	def _report_inst_alarm(self, instance: Instance, is_cleared: bool = False) -> None:
693	from motor.controller.observability.observability import Observability	731	from motor.controller.observability.observability import Observability
694		732
	@@ -731,14 +769,22 @@ class InstanceManager(ThreadSafeSingleton):
731	logger.warning(	769	logger.warning(
732	"No node managers found for instance %s(id:%d), setting to INACTIVE", instance.job_name, instance.id	770	"No node managers found for instance %s(id:%d), setting to INACTIVE", instance.job_name, instance.id
733	)	771	)
734	- return True	772	+ return True, []
735		773
		774	+ # Probe ALL node managers before returning — an early exit on the first
		775	+ # abnormal one would drop the still-reachable survivors from
		776	+ # `reachable_nms` and the partial-loss shutdown would miss them.
		777	+ reachable_nms = []
		778	+ has_abnormal = False
736	for node_mgr in node_managers:	779	for node_mgr in node_managers:
737	try:	780	try:
738	response = NodeManagerApiClient.query_status(node_mgr)	781	response = NodeManagerApiClient.query_status(node_mgr)
739	if isinstance(response, dict) and "status" in response:	782	if isinstance(response, dict) and "status" in response:
740	is_normal = response.get("status", False)	783	is_normal = response.get("status", False)
741	- if not is_normal:	784	+ if is_normal:
		785	+ reachable_nms.append(node_mgr)
		786	+ else:
		787	+ has_abnormal = True
742	logger.warning(	788	logger.warning(
743	"Node manager %s:%s reports abnormal endpoints for instance %s(id:%d)",	789	"Node manager %s:%s reports abnormal endpoints for instance %s(id:%d)",
744	node_mgr.pod_ip,	790	node_mgr.pod_ip,
	@@ -746,8 +792,8 @@ class InstanceManager(ThreadSafeSingleton):
746	instance.job_name,	792	instance.job_name,
747	instance.id,	793	instance.id,
748	)	794	)
749	- return True
750	else:	795	else:
		796	+ has_abnormal = True
751	logger.warning(	797	logger.warning(
752	"Invalid response from node manager %s:%s for instance %s(id:%d): %s",	798	"Invalid response from node manager %s:%s for instance %s(id:%d): %s",
753	node_mgr.pod_ip,	799	node_mgr.pod_ip,
	@@ -756,8 +802,8 @@ class InstanceManager(ThreadSafeSingleton):
756	instance.id,	802	instance.id,
757	response,	803	response,
758	)	804	)
759	- return True
760	except Exception as e:	805	except Exception as e:
		806	+ has_abnormal = True
761	logger.warning(	807	logger.warning(
762	"Failed to check node manager %s:%s status for instance %s(id:%d): %s",	808	"Failed to check node manager %s:%s status for instance %s(id:%d): %s",
763	node_mgr.pod_ip,	809	node_mgr.pod_ip,
	@@ -766,10 +812,11 @@ class InstanceManager(ThreadSafeSingleton):
766	instance.id,	812	instance.id,
767	e,	813	e,
768	)	814	)
769	- return True
770		815
		816	+ if has_abnormal:
		817	+ return True, reachable_nms
771	logger.info("All node managers report normal status for instance %s(id:%d)", instance.job_name, instance.id)	818	logger.info("All node managers report normal status for instance %s(id:%d)", instance.job_name, instance.id)
772	- return False	819	+ return False, reachable_nms
773		820
774	def _handle_deleted(self, from_state: InsStatus, condition_event: InsConditionEvent, instance: Instance) -> None:	821	def _handle_deleted(self, from_state: InsStatus, condition_event: InsConditionEvent, instance: Instance) -> None:
775	if from_state == InsStatus.DELETED:	822	if from_state == InsStatus.DELETED:
	@@ -802,6 +849,18 @@ class InstanceManager(ThreadSafeSingleton):
802	event = InsConditionEvent.INSTANCE_PAUSED	849	event = InsConditionEvent.INSTANCE_PAUSED
803	to_state = self.transitions.get((from_state, event), None)	850	to_state = self.transitions.get((from_state, event), None)
804	elif instance.is_all_endpoints_ready():	851	elif instance.is_all_endpoints_ready():
		852	+ if not instance.is_all_endpoints_heartbeat_fresh():
		853	+ # Every endpoint reports NORMAL but some endpoint's heartbeat
		854	+ # timed out (its NodeManager is gone and nobody updates its
		855	+ # status field anymore). A lone surviving NodeManager's
		856	+ # heartbeat must not flip the instance ACTIVE — the
		857	+ # management loop owns the timeout flow, so leave the state
		858	+ # untouched this round.
		859	+ logger.debug(
		860	+ "Instance %d ready by status but has stale heartbeats; skipping state transition",
		861	+ instance.id,
		862	+ )
		863	+ return True
805	event = InsConditionEvent.INSTANCE_NORMAL	864	event = InsConditionEvent.INSTANCE_NORMAL
806	to_state = self.transitions.get((from_state, event), None)	865	to_state = self.transitions.get((from_state, event), None)
807	elif instance.is_have_one_endpoint_abnormal():	866	elif instance.is_have_one_endpoint_abnormal():

Mmotor/controller/fault_tolerance/fault_manager.py+25-5

	@@ -534,11 +534,26 @@ class FaultManager(_PersistenceMixin, _ResourceManagerMixin, ThreadSafeSingleton
534	current_level = ins_metadata.strategy_fault_level	534	current_level = ins_metadata.strategy_fault_level
535	current_cls_name = current_strategy.__class__.__name__ if current_strategy else None	535	current_cls_name = current_strategy.__class__.__name__ if current_strategy else None
536		536
537	- new_strategy_cls = (	537	+ # Escalation chain: when the previous strategy finished without
538	- self.strategies[fault_level](fault_code, ins_id, self.config)	538	+ # restoring health, run the fallback (EngineRelaunchStrategy —
539	- if fault_level != FaultLevel.HEALTHY	539	+ # restart engines, then containers) instead of re-running the
540	- else None	540	+ # same strategy in a loop. Honours the enable_engine_relaunch
541	- )	541	+ # switch like the level2_strategy hook: disabling it restores the
		542	+ # legacy behavior (heartbeat suicide -> k8s pod restart only).
		543	+ if (
		544	+ fault_level != FaultLevel.HEALTHY
		545	+ and ins_metadata.prev_strategy_failed
		546	+ and self.config.fault_tolerance_config.enable_engine_relaunch
		547	+ ):
		548	+ from motor.controller.fault_tolerance.strategy import EngineRelaunchStrategy
		549	+
		550	+ new_strategy_cls = EngineRelaunchStrategy
		551	+ else:
		552	+ new_strategy_cls = (
		553	+ self.strategies[fault_level](fault_code, ins_id, self.config)
		554	+ if fault_level != FaultLevel.HEALTHY
		555	+ else None
		556	+ )
542		557
543	if new_strategy_cls is not None:	558	if new_strategy_cls is not None:
544	should_switch = False	559	should_switch = False
	@@ -584,6 +599,11 @@ class FaultManager(_PersistenceMixin, _ResourceManagerMixin, ThreadSafeSingleton
584	ins_metadata.strategy.__class__.__name__,	599	ins_metadata.strategy.__class__.__name__,
585	ins_metadata.strategy_fault_level.name,	600	ins_metadata.strategy_fault_level.name,
586	)	601	)
		602	+ # Escalation bookkeeping: a failed strategy leaves the
		603	+ # software faults in place (so the fault level stays) and
		604	+ # marks the instance for the fallback strategy; a
		605	+ # successful one clears the marker.
		606	+ ins_metadata.prev_strategy_failed = ins_metadata.strategy.is_failed()
587	ins_metadata.strategy = None	607	ins_metadata.strategy = None
588	ins_metadata.strategy_fault_level = FaultLevel.HEALTHY	608	ins_metadata.strategy_fault_level = FaultLevel.HEALTHY
589	need_post_completion = True	609	need_post_completion = True

Mmotor/controller/fault_tolerance/fault_types.py+4-0

	@@ -260,6 +260,10 @@ class InstanceMetadata(BaseModel):
260	strategy_fault_level: FaultLevel = Field(	260	strategy_fault_level: FaultLevel = Field(
261	default=FaultLevel.HEALTHY, description="Fault level of the currently running strategy"	261	default=FaultLevel.HEALTHY, description="Fault level of the currently running strategy"
262	)	262	)
		263	+ #: Set when the last strategy finished without restoring health; the
		264	+ #: strategy center then escalates to the fallback strategy
		265	+ #: (EngineRelaunchStrategy) instead of re-running the same strategy.
		266	+ prev_strategy_failed: bool = Field(default=False, description="Last strategy failed to recover")
263		267
264	# Non-serializable fields (excluded from serialization)	268	# Non-serializable fields (excluded from serialization)
265	lock: Any = Field(default=None, exclude=True)	269	lock: Any = Field(default=None, exclude=True)

Mmotor/controller/fault_tolerance/mixin/resource_manager.py+3-1

	@@ -463,7 +463,9 @@ class _ResourceManagerMixin:
463		463
464	node_metadata.hardware_fault_infos.clear()	464	node_metadata.hardware_fault_infos.clear()
465	for code, infos in grouped.items():	465	for code, infos in grouped.items():
466	- info = max(infos, key=lambda i: i.fault_level.value)	466	+ # Copy before adjusting: the dynamic PreSeparateNPU downgrade
		467	+ # below must not mutate the caller's FaultInfo object.
		468	+ info = max(infos, key=lambda i: i.fault_level.value).model_copy()
467	info.fault_category = FaultCategory.HARDWARE	469	info.fault_category = FaultCategory.HARDWARE
468		470
469	# Dynamically adjust PreSeparateNPU fault level based on	471	# Dynamically adjust PreSeparateNPU fault level based on

Mmotor/controller/fault_tolerance/strategy/__init__.py+5-2

	@@ -1,4 +1,3 @@
1	-# -- coding: utf-8 --
2	# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.	1	# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
3	# MindIE is licensed under Mulan PSL v2.	2	# MindIE is licensed under Mulan PSL v2.
4	# You can use this software according to the terms and conditions of the Mulan PSL v2.	3	# You can use this software according to the terms and conditions of the Mulan PSL v2.
	@@ -8,6 +7,7 @@
8	# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,	7	# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
9	# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.	8	# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
10	# See the Mulan PSL v2 for more details.	9	# See the Mulan PSL v2 for more details.
		10	+
11	"""	11	"""
12	Fault tolerance strategy module - contains fault recovery strategies.	12	Fault tolerance strategy module - contains fault recovery strategies.
13	"""	13	"""
	@@ -17,8 +17,11 @@ __all__ = [
17	"generate_strategy_map",	17	"generate_strategy_map",
18	"ScaleP2DStrategy",	18	"ScaleP2DStrategy",
19	"TokenReinferenceStrategy",	19	"TokenReinferenceStrategy",
		20	+ "EngineRelaunchStrategy",
20	]	21	]
21		22
22	-from .strategy import StrategyBase, generate_strategy_map	23	+from .base import StrategyBase
		24	+from .strategy import generate_strategy_map
23	from .scale_p2d import ScaleP2DStrategy	25	from .scale_p2d import ScaleP2DStrategy
24	from .token_reinference import TokenReinferenceStrategy	26	from .token_reinference import TokenReinferenceStrategy
		27	+from .engine_relaunch import EngineRelaunchStrategy

Amotor/controller/fault_tolerance/strategy/base.py+53-0

@@ -0,0 +1,53 @@
	1	+# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
	2	+# MindIE is licensed under Mulan PSL v2.
	3	+# You can use this software according to the terms and conditions of the Mulan PSL v2.
	4	+# You may obtain a copy of Mulan PSL v2 at:
	5	+# http://license.coscl.org.cn/MulanPSL2
	6	+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
	7	+# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
	8	+# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
	9	+# See the Mulan PSL v2 for more details.
	10	+"""Strategy base class shared by all fault recovery strategies."""
	11	+
	12	+import threading
	13	+from abc import ABC, abstractmethod
	14	+
	15	+
	16	+class StrategyBase(ABC):
	17	+ """Strategy base class"""
	18	+
	19	+ def __init__(self) -> None:
	20	+ self.event = threading.Event()
	21	+ self.name = self.__class__.__name__
	22	+ self._is_finished = False
	23	+ self._is_failed = False
	24	+ self._lock = threading.Lock()
	25	+
	26	+ @abstractmethod
	27	+ def execute(self, instance_id: int):
	28	+ """
	29	+ Execute the strategy with the instance id.
	30	+ """
	31	+ raise NotImplementedError
	32	+
	33	+ @abstractmethod
	34	+ def stop(self) -> None:
	35	+ raise NotImplementedError
	36	+
	37	+ def is_finished(self) -> bool:
	38	+ with self._lock:
	39	+ return self._is_finished
	40	+
	41	+ def mark_failed(self) -> None:
	42	+ """Mark the strategy run as failed (recovery did not complete).
	43	+
	44	+ Any recovery strategy that finishes without restoring health calls
	45	+ this; the strategy center then escalates to the fallback strategy
	46	+ (EngineRelaunchStrategy) on the next round.
	47	+ """
	48	+ with self._lock:
	49	+ self._is_failed = True
	50	+
	51	+ def is_failed(self) -> bool:
	52	+ with self._lock:
	53	+ return self._is_failed

Amotor/controller/fault_tolerance/strategy/engine_relaunch.py+360-0

@@ -0,0 +1,360 @@
	1	+# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
	2	+# MindIE is licensed under Mulan PSL v2.
	3	+# You can use this software according to the terms and conditions of the Mulan PSL v2.
	4	+# You may obtain a copy of Mulan PSL v2 at:
	5	+# http://license.coscl.org.cn/MulanPSL2
	6	+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
	7	+# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
	8	+# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
	9	+# See the Mulan PSL v2 for more details.
	10	+"""EngineRelaunchStrategy: fallback recovery that restarts engines, then containers.
	11	+
	12	+The fallback of the recovery ladder: fast strategies (token reinference,
	13	+UCE, elastic scaling) try to recover without restarting the engine; when one
	14	+of them fails (``mark_failed``) — or an engine is reported DEAD outright —
	15	+this strategy takes over:
	16	+
	17	+1. Phase 1 (relaunch engines): dispatch ``/node-manager/engine-restart``
	18	+ to every NodeManager of the instance so all ranks restart in the same
	19	+ window (cross-machine collective-group consistency), then poll
	20	+ ``/node-manager/status`` until every NodeManager reports all endpoints
	21	+ NORMAL again.
	22	+2. Phase 2 (restart containers): if relaunching engines fails (dispatch
	23	+ error, a NodeManager unreachable, or the completion timeout elapsed),
	24	+ unfreeze each reachable NodeManager's suicide counter (``abort``) so the
	25	+ heartbeat mechanism restarts the pod via k8s. The instance's NodeManagers
	26	+ either all relaunch their engines or all restart their containers — a
	27	+ partial mix would leave the collective group split.
	28	+
	29	+The instance briefly goes INACTIVE while heartbeats report ABNORMAL and
	30	+returns to ACTIVE automatically once the engines report NORMAL again
	31	+(``INACTIVE + INSTANCE_NORMAL -> ACTIVE`` in the instance state machine).
	32	+"""
	33	+
	34	+import threading
	35	+import time
	36	+from dataclasses import dataclass, field
	37	+from enum import Enum
	38	+
	39	+from motor.common.logger import get_logger
	40	+from motor.common.resources import InsStatus
	41	+from motor.controller.api_client.node_manager_api_client import NodeManagerApiClient
	42	+from motor.controller.fault_tolerance.strategy.base import StrategyBase
	43	+
	44	+logger = get_logger(__name__)
	45	+
	46	+
	47	+class RelaunchState(str, Enum):
	48	+ """States of the engine relaunch workflow."""
	49	+
	50	+ INIT = "init"
	51	+ RELAUNCH_ENGINE = "relaunch_engine"
	52	+ RESTART_CONTAINER = "restart_container"
	53	+ SUCCESS = "success"
	54	+ FAILED = "failed"
	55	+
	56	+
	57	+@dataclass
	58	+class RelaunchContext:
	59	+ """Mutable context carried through a single relaunch run."""
	60	+
	61	+ instance_id: int
	62	+ current_state: RelaunchState = RelaunchState.INIT
	63	+ node_managers: list = field(default_factory=list)
	64	+ #: Per-NodeManager restart-dispatch outcome (pod_ip -> True when the
	65	+ #: restart command was accepted). Phase 2 only aborts the ones that were
	66	+ #: NOT dispatched — a NodeManager that is already relaunching keeps its
	67	+ #: freeze (its deadline provides the fallback) instead of being torn down
	68	+ #: mid-relaunch.
	69	+ dispatch_results: dict = field(default_factory=dict)
	70	+ last_error: str \| None = None
	71	+ start_time: float = field(default_factory=time.time)
	72	+
	73	+
	74	+class EngineRelaunchStrategy(StrategyBase):
	75	+ """Fallback recovery: relaunch all engines in place, then restart containers.
	76	+
	77	+ The strategy center escalates here when (a) an engine is reported DEAD
	78	+ (ENGINE_DEAD L2 fault) or (b) any previous strategy finished with
	79	+ ``mark_failed()`` (see ``fault_manager._process_instance_strategy``).
	80	+ """
	81	+
	82	+ DISPATCH_RETRY_BACKOFF_SEC = 2.0
	83	+
	84	+ def __init__(self) -> None:
	85	+ super().__init__()
	86	+ self.context: RelaunchContext \| None = None
	87	+ self.engine_relaunch_poll_interval_sec = self._resolve_config("engine_relaunch_poll_interval_sec", 5.0)
	88	+ self.engine_relaunch_complete_timeout_sec = self._resolve_config("engine_relaunch_complete_timeout_sec", 600)
	89	+ self.engine_relaunch_dispatch_retries = self._resolve_config("engine_relaunch_dispatch_retries", 3)
	90	+ self.engine_relaunch_nm_unreachable_threshold = self._resolve_config(
	91	+ "engine_relaunch_nm_unreachable_threshold", 3
	92	+ )
	93	+
	94	+ @staticmethod
	95	+ def _resolve_config(attr: str, default: float) -> float:
	96	+ """Read a relaunch knob from the FaultManager config; fall back on errors."""
	97	+ # Local import to avoid a circular dependency:
	98	+ # fault_manager → strategy/__init__ → engine_relaunch → fault_manager
	99	+ from motor.controller.fault_tolerance.fault_manager import FaultManager # pylint: disable=cyclic-import
	100	+
	101	+ try:
	102	+ ft_config = FaultManager().config.fault_tolerance_config
	103	+ return getattr(ft_config, attr, default)
	104	+ except Exception as e:
	105	+ logger.warning("Failed to resolve relaunch config %s, using default %s: %s", attr, default, e)
	106	+ return default
	107	+
	108	+ # ------------------------------------------------------------------
	109	+ # execute: two-phase recovery
	110	+ # ------------------------------------------------------------------
	111	+
	112	+ def execute(self, instance_id: int) -> None:
	113	+ self.context = RelaunchContext(instance_id=instance_id)
	114	+ try:
	115	+ instance = self._get_instance(instance_id)
	116	+ if instance is None:
	117	+ logger.error("Engine relaunch aborted: instance %d not found", instance_id)
	118	+ self._finish(RelaunchState.FAILED, "instance not found")
	119	+ return
	120	+
	121	+ self.context.node_managers = list(instance.get_node_managers())
	122	+ if not self.context.node_managers:
	123	+ logger.error("Engine relaunch aborted: instance %d has no node managers", instance_id)
	124	+ self._finish(RelaunchState.FAILED, "no node managers")
	125	+ return
	126	+
	127	+ if not self._phase_relaunch_engine(instance_id):
	128	+ self._phase_restart_container()
	129	+ return
	130	+
	131	+ self._finish(RelaunchState.SUCCESS)
	132	+ except Exception as e:
	133	+ logger.exception("Engine relaunch failed for instance %d", instance_id)
	134	+ self._finish(RelaunchState.FAILED, str(e))
	135	+
	136	+ def _phase_relaunch_engine(self, instance_id: int) -> bool:
	137	+ """Phase 1: probe all NodeManagers, dispatch restart, poll recovery.
	138	+
	139	+ Returns True when every NodeManager reports all endpoints NORMAL
	140	+ within the completion timeout; False escalates to Phase 2.
	141	+ """
	142	+ self.context.current_state = RelaunchState.RELAUNCH_ENGINE
	143	+ if self._any_event_set():
	144	+ return False
	145	+
	146	+ # 1. Probe every NodeManager first: dispatching a restart to a dead
	147	+ # NodeManager is pointless, and its pod restart would split the
	148	+ # collective group — escalate to container restart instead.
	149	+ self.context.dispatch_results = self._dispatch_to_all("restart", probe_first=True)
	150	+ if self.context.dispatch_results is None:
	151	+ return False
	152	+
	153	+ # 2. Poll until all NodeManagers report all endpoints NORMAL.
	154	+ unreachable_counts: dict = {}
	155	+ while not self._any_event_set():
	156	+ if time.time() - self.context.start_time > self.engine_relaunch_complete_timeout_sec:
	157	+ logger.error(
	158	+ "Engine relaunch timeout after %ss for instance %d",
	159	+ self.engine_relaunch_complete_timeout_sec,
	160	+ instance_id,
	161	+ )
	162	+ return False
	163	+
	164	+ instance = self._get_instance(instance_id)
	165	+ if instance is None or instance.status == InsStatus.DELETED:
	166	+ logger.info("Instance %d gone during engine relaunch, finishing", instance_id)
	167	+ return True
	168	+
	169	+ all_normal = True
	170	+ for node_mgr in list(instance.get_node_managers()):
	171	+ try:
	172	+ # relaxed: "no endpoint ABNORMAL" — a freshly relaunched
	173	+ # engine reports INITIAL while loading its model, which
	174	+ # counts as recovering. Waiting for full NORMAL here would
	175	+ # make the completion timeout fight the model load time.
	176	+ response = NodeManagerApiClient.query_status(node_mgr, relaxed=True)
	177	+ unreachable_counts.pop(node_mgr.pod_ip, None)
	178	+ if not response.get("status"):
	179	+ all_normal = False
	180	+ except Exception as e:
	181	+ unreachable_counts[node_mgr.pod_ip] = unreachable_counts.get(node_mgr.pod_ip, 0) + 1
	182	+ all_normal = False
	183	+ if unreachable_counts[node_mgr.pod_ip] >= self.engine_relaunch_nm_unreachable_threshold:
	184	+ logger.error(
	185	+ "NodeManager %s unreachable for %d consecutive polls during engine relaunch",
	186	+ node_mgr.pod_ip,
	187	+ unreachable_counts[node_mgr.pod_ip],
	188	+ )
	189	+ return False
	190	+ logger.debug("NodeManager %s not ready yet: %s", node_mgr.pod_ip, e)
	191	+
	192	+ if all_normal:
	193	+ logger.info(
	194	+ "Engine relaunch finished: all endpoints of instance %d recovering (no ABNORMAL)", instance_id
	195	+ )
	196	+ return True
	197	+
	198	+ if self.event.wait(self.engine_relaunch_poll_interval_sec):
	199	+ return False
	200	+
	201	+ return False
	202	+
	203	+ def _phase_restart_container(self) -> None:
	G G ganglv12 天前严重程度：建议问题： Phase 2 的 abort 发给 context 中全部 NM（含 Phase 1 已成功接收 restart、正在重拉引擎的 NM），与 docstring 宣称的“要么全部重拉、要么全部重启容器”不一致。原因：当部分 NM 派发失败（如瞬时不达）时：NM#1 已接收 restart 开始重拉（720s 冻结），NM#2 派发失败 → Phase 2 对两者都发 abort → NM#1 的冻结被取消。虽然引擎 INITIAL/NORMAL 时 abort 无害、最终能收敛，但若 NM#1 模型加载失败转为 ABNORMAL，它会在无冻结下 15s 提前自杀，形成文档明言要避免的“部分重拉 + 部分重启”混合窗口，集合通信组被撕开。怎么改：在 RelaunchContext 中记录每个 NM 的 dispatch 结果，Phase 2 只对派发失败的 NM 发 abort；对已成功派发的 NM 保留其冻结（deadline 到期后自然兜底）。 jason lyu 12 天前评论： 12 天前评论： jason lyu 12 天前评论： 12 天前评论：
	204	+ """Phase 2: restart containers — abort the NodeManagers that were not
	205	+ relaunching.
	206	+
	207	+ Only the NodeManagers whose restart dispatch FAILED (or that were
	208	+ never reached) get an abort: their engines were never killed, so the
	209	+ heartbeat mechanism must restart the container via k8s. A NodeManager
	210	+ that already accepted the restart keeps its suicide freeze — it is
	211	+ relaunching in place and its freeze deadline provides the eventual
	212	+ fallback. Aborting it mid-relaunch would tear down an engine that is
	213	+ actually recovering and split the collective group.
	214	+ """
	215	+ self.context.current_state = RelaunchState.RESTART_CONTAINER
	216	+ logger.warning(
	217	+ "Engine relaunch failed for instance %d, escalating to container restart",
	218	+ self.context.instance_id,
	219	+ )
	220	+ for node_mgr in self.context.node_managers:
	221	+ if (self.context.dispatch_results or {}).get(node_mgr.pod_ip, False):
	222	+ logger.info(
	223	+ "Node manager %s already relaunching (restart dispatched), keeping its freeze", node_mgr.pod_ip
	224	+ )
	225	+ continue
	226	+ if NodeManagerApiClient.restart_engine(node_mgr, action="abort", instance_id=self.context.instance_id):
	wangyang12 天前 abort 发送失败时没日志也没 else，哪个 NM 的容器兜底没生效完全不可见。 jason lyu 8 天前评论： 8 天前评论：
	227	+ logger.info("Abort sent to node manager %s: suicide counter unfrozen", node_mgr.pod_ip)
	228	+ else:
	229	+ logger.error(
	230	+ "Abort to node manager %s failed: its container-restart fallback may not kick in",
	231	+ node_mgr.pod_ip,
	232	+ )
	233	+ self._finish(RelaunchState.FAILED, "engine relaunch failed, escalated to container restart")
	234	+
	235	+ def _dispatch_to_all(self, action: str, probe_first: bool = False) -> dict \| None:
	236	+ """Send the restart (or abort) command to every NodeManager.
	237	+
	238	+ Returns the per-NodeManager dispatch outcome (``{pod_ip: bool}``),
	239	+ or None when the probe found a NodeManager unreachable (nothing was
	240	+ dispatched — the whole instance escalates for consistency).
	241	+ """
	242	+ node_managers = list(self.context.node_managers)
	243	+ if probe_first:
	244	+ node_managers = self._probe_reachable(node_managers)
	245	+ if node_managers is None:
	246	+ return None
	247	+
	248	+ results: dict = {}
	249	+ for node_mgr in node_managers:
	250	+ success = False
	251	+ for attempt in range(1, self.engine_relaunch_dispatch_retries + 1):
	G G ganglv12 天前严重程度：严重问题： NM 端引擎重启进行中返回 409 被当作派发失败，重试后误升级为容器重启。原因： NM 端 `/node-manager/engine-restart` 路由在 `_engine_restart_in_progress` 期间返回 409，而 SafeHTTPSClient 对非 2xx 会 raise（raise_for_status），restart_engine 返回 False。这里把 409 与“NM 不可达/拒绝”同等对待，重试 3 次（每次 2s backoff）后 `dispatched_all=False` → `_phase_restart_container` 对全部 NM 发 abort。而 NM 端 restart 全程持有 `_engine_restart_in_progress`（`_kill_process_tree` 的 `_wait_group_exit` 最多等 5s + pull spawn），很可能超过 2s 重试间隔——即引擎正在成功重拉时，Controller 却升级为容器重启：已成功重启的 NM 的 720s 冻结被 `unfreeze_suicide` 取消，模型加载失败时该 NM 会在无冻结下 15s 提前自杀，破坏 PR 核心目标“容器不重启”。怎么改：对 409 单独处理：视为“该 NM 已在重启中”，跳过重试、标记该 NM 派发成功，进入轮询等待 NORMAL 即可；或 NM 端把重启进行中改为 200/202（重启本身是幂等的）。 `if not success: # 例如按响应状态码特判 409（进行中） success = already_in_progress # 已在重启中，无需重复派发` jason lyu 12 天前评论： 12 天前评论： jason lyu 12 天前评论： 12 天前评论：
	252	+ if self._any_event_set():
	253	+ return results
	254	+ success = NodeManagerApiClient.restart_engine(
	255	+ node_mgr, action=action, instance_id=self.context.instance_id
	256	+ )
	257	+ if success:
	258	+ break
	259	+ logger.warning(
	260	+ "Dispatch %s to node manager %s failed (attempt %d/%d)",
	261	+ action,
	262	+ node_mgr.pod_ip,
	263	+ attempt,
	264	+ self.engine_relaunch_dispatch_retries,
	265	+ )
	266	+ if attempt < self.engine_relaunch_dispatch_retries:
	267	+ if self.event.wait(self.DISPATCH_RETRY_BACKOFF_SEC):
	268	+ return results
	269	+ results[node_mgr.pod_ip] = success
	270	+ return results
	271	+
	272	+ def _probe_reachable(self, node_managers: list) -> list \| None:
	273	+ """Return the reachable subset after retries; None when any is unreachable."""
	274	+ reachable = []
	275	+ for node_mgr in node_managers:
	276	+ probed = False
	277	+ for attempt in range(1, self.engine_relaunch_dispatch_retries + 1):
	278	+ if self._any_event_set():
	279	+ return None
	280	+ try:
	281	+ NodeManagerApiClient.query_status(node_mgr)
	282	+ probed = True
	283	+ break
	284	+ except Exception as e:
	285	+ logger.warning(
	286	+ "NodeManager %s unreachable during probe (attempt %d/%d): %s",
	287	+ node_mgr.pod_ip,
	288	+ attempt,
	289	+ self.engine_relaunch_dispatch_retries,
	290	+ e,
	291	+ )
	292	+ if attempt < self.engine_relaunch_dispatch_retries:
	293	+ if self.event.wait(self.DISPATCH_RETRY_BACKOFF_SEC):
	294	+ return None
	295	+ if not probed:
	296	+ logger.error(
	297	+ "NodeManager %s unreachable — escalating instance %d to container restart",
	298	+ node_mgr.pod_ip,
	299	+ self.context.instance_id,
	300	+ )
	301	+ return None
	302	+ reachable.append(node_mgr)
	303	+ return reachable
	304	+
	305	+ # ------------------------------------------------------------------
	306	+ # helpers
	307	+ # ------------------------------------------------------------------
	308	+
	309	+ def _any_event_set(self) -> bool:
	310	+ return self.event.is_set()
	311	+
	312	+ @staticmethod
	313	+ def _get_instance(instance_id: int):
	314	+ """InstanceManager lookup; lazy import to keep the import graph acyclic."""
	315	+ from motor.controller.core.instance_manager import InstanceManager
	316	+
	317	+ return InstanceManager().get_instance(instance_id)
	318	+
	319	+ def _finish(self, state: RelaunchState, error: str \| None = None) -> None:
	320	+ if self.context is not None:
	321	+ self.context.current_state = state
	322	+ self.context.last_error = error
	323	+ if state == RelaunchState.FAILED:
	324	+ self.mark_failed()
	325	+ logger.error(
	326	+ "Engine relaunch failed for instance %s after %.1fs: %s",
	327	+ self.context.instance_id if self.context else "?",
	328	+ time.time() - self.context.start_time if self.context else 0.0,
	329	+ error,
	330	+ )
	331	+ else:
	332	+ logger.info(
	333	+ "Engine relaunch succeeded for instance %s after %.1fs",
	334	+ self.context.instance_id if self.context else "?",
	335	+ time.time() - self.context.start_time if self.context else 0.0,
	336	+ )
	337	+ with self._lock:
	338	+ self._is_finished = True
	339	+
	340	+ def stop(self) -> None:
	341	+ """Interrupt the strategy (upgrade/switch); unfreeze the NodeManagers.
	342	+
	343	+ The suicide counters were frozen for up to ``engine_restart_freeze_sec``
	344	+ on every NodeManager — unfreeze them so the container-restart
	345	+ fallback is not delayed for the whole freeze window.
	346	+ """
	347	+ logger.info("Engine relaunch strategy stopped.")
	348	+ self.event.set()
	349	+ if self.context is None:
	350	+ return
	351	+ node_managers = list(self.context.node_managers)
	352	+
	353	+ def _unfreeze_async() -> None:
	354	+ for node_mgr in node_managers:
	355	+ try:
	356	+ NodeManagerApiClient.restart_engine(node_mgr, action="abort", instance_id=self.context.instance_id)
	357	+ except Exception as e:
	358	+ logger.warning("Failed to abort relaunch on %s: %s", node_mgr.pod_ip, e)
	359	+
	360	+ threading.Thread(target=_unfreeze_async, daemon=True, name="engine_relaunch_abort").start()

Mmotor/controller/fault_tolerance/strategy/scale_p2d.py+3-3

Mmotor/controller/fault_tolerance/strategy/strategy.py+13-32

	@@ -7,36 +7,10 @@
7	# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,	7	# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
8	# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.	8	# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
9	# See the Mulan PSL v2 for more details.	9	# See the Mulan PSL v2 for more details.
10	-from abc import ABC, abstractmethod
11	-import threading
12	from collections.abc import Callable	10	from collections.abc import Callable
13	from motor.config.controller import ControllerConfig	11	from motor.config.controller import ControllerConfig
14	-from motor.controller.fault_tolerance.fault_types import FaultLevel	12	+from motor.controller.fault_tolerance.fault_types import FaultLevel, SpecialFaultCode
15	-	13	+from motor.controller.fault_tolerance.strategy.base import StrategyBase
16	-
17	-class StrategyBase(ABC):
18	- """Strategy base class"""
19	-
20	- def __init__(self) -> None:
21	- self.event = threading.Event()
22	- self.name = self.__class__.__name__
23	- self._is_finished = False
24	- self._lock = threading.Lock()
25	-
26	- @abstractmethod
27	- def execute(self, instance_id: int):
28	- """
29	- Execute the strategy with the instance id.
30	- """
31	- raise NotImplementedError
32	-
33	- @abstractmethod
34	- def stop(self) -> None:
35	- raise NotImplementedError
36	-
37	- def is_finished(self) -> bool:
38	- with self._lock:
39	- return self._is_finished
40		14
41		15
42	def healthy_strategy(fault_code: int, instance_id: int, config: ControllerConfig) -> type[StrategyBase] \| None:	16	def healthy_strategy(fault_code: int, instance_id: int, config: ControllerConfig) -> type[StrategyBase] \| None:
	@@ -49,15 +23,22 @@ def level1_strategy(fault_code: int, instance_id: int, config: ControllerConfig)
49		23
50		24
51	def level2_strategy(fault_code: int, instance_id: int, config: ControllerConfig) -> type[StrategyBase] \| None:	25	def level2_strategy(fault_code: int, instance_id: int, config: ControllerConfig) -> type[StrategyBase] \| None:
52	- # Software fault strategies (ENGINE_DEAD, ENGINE_UNHEALTHY) not implemented yet.	26	+ # A dead engine cannot be recovered in place — run the fallback relaunch
53	- # Software faults still update the instance fault level, but no automatic recovery.	27	+ # (restart engines, then containers). ENGINE_UNHEALTHY keeps the current
		28	+ # no-op behavior (fast-recovery strategies take it over once merged).
		29	+ if fault_code == int(SpecialFaultCode.ENGINE_DEAD):
		30	+ if not config.fault_tolerance_config.enable_engine_relaunch:
		31	+ return None
		32	+ from motor.controller.fault_tolerance.strategy.engine_relaunch import EngineRelaunchStrategy
		33	+
		34	+ return EngineRelaunchStrategy
54		35
55	# Hardware L2 faults: only handle whitelisted fault codes (token reinference)	36	# Hardware L2 faults: only handle whitelisted fault codes (token reinference)
56	if not config.fault_tolerance_config.enable_token_reinference:	37	if not config.fault_tolerance_config.enable_token_reinference:
57	return None	38	return None
58		39
59	if fault_code in [0x00F1FEF5, 0x08520003]:	40	if fault_code in [0x00F1FEF5, 0x08520003]:
60	- from motor.controller.fault_tolerance.strategy import TokenReinferenceStrategy	41	+ from motor.controller.fault_tolerance.strategy.token_reinference import TokenReinferenceStrategy
61		42
62	return TokenReinferenceStrategy	43	return TokenReinferenceStrategy
63	return None	44	return None
	@@ -74,7 +55,7 @@ def level4_strategy(fault_code: int, instance_id: int, config: ControllerConfig)
74	return None	55	return None
75		56
76	from motor.controller.core.instance_manager import InstanceManager	57	from motor.controller.core.instance_manager import InstanceManager
77	- from motor.controller.fault_tolerance.strategy import ScaleP2DStrategy	58	+ from motor.controller.fault_tolerance.strategy.scale_p2d import ScaleP2DStrategy
78		59
79	instance = InstanceManager().get_instance(instance_id)	60	instance = InstanceManager().get_instance(instance_id)
80	if instance is not None and instance.role == "decode":	61	if instance is not None and instance.role == "decode":

Mmotor/controller/fault_tolerance/strategy/token_reinference.py+3-7

	@@ -1,4 +1,3 @@
1	-# -- coding: utf-8 --
2	# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.	1	# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
3	# MindIE is licensed under Mulan PSL v2.	2	# MindIE is licensed under Mulan PSL v2.
4	# You can use this software according to the terms and conditions of the Mulan PSL v2.	3	# You can use this software according to the terms and conditions of the Mulan PSL v2.
	@@ -10,19 +9,16 @@
10	# See the Mulan PSL v2 for more details.	9	# See the Mulan PSL v2 for more details.
11		10
12	from motor.common.logger import get_logger	11	from motor.common.logger import get_logger
13	-from motor.controller.fault_tolerance.strategy import StrategyBase	12	+from motor.controller.fault_tolerance.strategy.base import StrategyBase
14		13
15		14
16	logger = get_logger(__name__)	15	logger = get_logger(__name__)
17		16
18		17
19	class TokenReinferenceStrategy(StrategyBase):	18	class TokenReinferenceStrategy(StrategyBase):
20	- def __init__(self) -> None:
21	- super().__init__()
22	-
23	def execute(self, instance_id: int) -> None:	19	def execute(self, instance_id: int) -> None:
24	while not self.event.is_set():	20	while not self.event.is_set():
25	pass	21	pass
26	-	22	+
27	def stop(self) -> None:	23	def stop(self) -> None:
28	- logger.info("Token Reinference strategy can not be stopped.")	24	+ logger.info("Token Reinference strategy can not be stopped.")

Mmotor/node_manager/api_server/node_manager_api.py+105-15

	@@ -11,6 +11,7 @@
11	import asyncio	11	import asyncio
12	import json	12	import json
13	import os	13	import os
		14	+import signal
14	import socket	15	import socket
15	import logging	16	import logging
16	import threading	17	import threading
	@@ -25,8 +26,8 @@ from motor.config.node_manager import NodeManagerConfig
25	from motor.node_manager.core.heartbeat_manager import HeartbeatManager	26	from motor.node_manager.core.heartbeat_manager import HeartbeatManager
26	from motor.common.logger import ApiAccessFilter, get_logger	27	from motor.common.logger import ApiAccessFilter, get_logger
27	from motor.common.resources.http_msg_spec import StartCmdMsg	28	from motor.common.resources.http_msg_spec import StartCmdMsg
28	-from motor.node_manager.core.engine_manager import EngineManager	29	+from motor.node_manager.core.register_manager import RegisterManager
29	-from motor.node_manager.core.daemon import Daemon	30	+from motor.node_manager.core.daemon import Daemon, EngineRestartInProgressError, EngineRestartParamError
30	from motor.node_manager.core.api_ready_event import clear_api_ready, mark_api_ready, wait_until_api_ready	31	from motor.node_manager.core.api_ready_event import clear_api_ready, mark_api_ready, wait_until_api_ready
31	from motor.common.resources.instance import PDRole	32	from motor.common.resources.instance import PDRole
32	from motor.common.utils.snapshot_utils import is_restored_from_host_side_snapshot	33	from motor.common.utils.snapshot_utils import is_restored_from_host_side_snapshot
	@@ -57,11 +58,11 @@ async def start_instance(request: Request):
57	try:	58	try:
58	payload = await request.json()	59	payload = await request.json()
59	start_msg = StartCmdMsg(**payload)	60	start_msg = StartCmdMsg(**payload)
60	- engine_manager = EngineManager()	61	+ register_manager = RegisterManager()
61		62
62	async with thread_semaphore:	63	async with thread_semaphore:
63	try:	64	try:
64	- parsed_ok = await asyncio.to_thread(engine_manager.parse_start_cmd, start_msg)	65	+ parsed_ok = await asyncio.to_thread(register_manager.parse_start_cmd, start_msg)
65	except Exception as inner_err:	66	except Exception as inner_err:
66	logger.error("Failed to parse start command: %s", inner_err)	67	logger.error("Failed to parse start command: %s", inner_err)
67	raise HTTPException(	68	raise HTTPException(
	@@ -77,13 +78,13 @@ async def start_instance(request: Request):
77	# Use start_msg.master_dp_ip to update snapshot metadata for engine resume	78	# Use start_msg.master_dp_ip to update snapshot metadata for engine resume
78	# Update endpoint and set started after restore flag	79	# Update endpoint and set started after restore flag
79	if is_restored_from_host_side_snapshot():	80	if is_restored_from_host_side_snapshot():
80	- await asyncio.to_thread(EngineManager().engine_resume_prepare, start_msg)	81	+ await asyncio.to_thread(RegisterManager().engine_resume_prepare, start_msg)
81	HeartbeatManager().update_endpoint(start_msg)	82	HeartbeatManager().update_endpoint(start_msg)
82	HeartbeatManager().set_started_after_restore(True)	83	HeartbeatManager().set_started_after_restore(True)
83	return {}	84	return {}
84		85
85	# If snapshot mode is not disabled, prepare snapshot runtime directories and metadata file for engine suspend	86	# If snapshot mode is not disabled, prepare snapshot runtime directories and metadata file for engine suspend
86	- await asyncio.to_thread(EngineManager().engine_suspend_prepare)	87	+ await asyncio.to_thread(RegisterManager().engine_suspend_prepare)
87		88
88	daemon = Daemon()	89	daemon = Daemon()
89	try:	90	try:
	@@ -93,7 +94,7 @@ async def start_instance(request: Request):
93	start_msg.endpoints,	94	start_msg.endpoints,
94	start_msg.instance_id,	95	start_msg.instance_id,
95	start_msg.master_dp_ip,	96	start_msg.master_dp_ip,
96	- engine_manager.d2d_peer_ips,	97	+ register_manager.d2d_peer_ips,
97	start_msg.node_rank,	98	start_msg.node_rank,
98	)	99	)
99	except Exception as pull_err:	100	except Exception as pull_err:
	@@ -113,8 +114,9 @@ async def start_instance(request: Request):
113	) from ls_err	114	) from ls_err
114		115
115	HeartbeatManager().update_endpoint(start_msg)	116	HeartbeatManager().update_endpoint(start_msg)
116	- HeartbeatManager().start()	117	+ # Gate the engine-status probing on the Daemon's engine-ready handoff
117	- engine_manager.start()	118	+ # (mgmt ports up) — the HeartbeatManager has no readiness logic.
		119	+ HeartbeatManager().start(engine_ready_event=daemon.engine_ready_event)
118	return {}	120	return {}
119		121
120	except HTTPException as http_err:	122	except HTTPException as http_err:
	@@ -127,13 +129,27 @@ async def start_instance(request: Request):
127	) from err	129	) from err
128		130
129		131
		132	+def _self_terminate() -> None:
		133	+ """SIGTERM this process: the Application treats it as a graceful shutdown
		134	+ and exits (-1) — k8s then restarts the pod.
		135	+ """
		136	+ os.kill(os.getpid(), signal.SIGTERM)
		137	+
		138	+
130	@app.post("/node-manager/stop")	139	@app.post("/node-manager/stop")
131	async def stop_instance(request: Request):	140	async def stop_instance(request: Request):
132	- """	141	+ """Stop all engine processes, then terminate this NodeManager.
133	- Stop all engine processes by invoking Daemon.exit_daemon().	142	+
		143	+ The Controller dispatches ``/node-manager/stop`` as the "suicide"
		144	+ instruction: engine stop followed by process exit (-1), which k8s turns
		145	+ into a pod restart. Used for instance teardown and for partial-loss
		146	+ coordination (the surviving NodeManagers of a cross-machine instance exit
		147	+ so the whole instance restarts together).
134	"""	148	"""
135	try:	149	try:
136	await asyncio.to_thread(Daemon().stop)	150	await asyncio.to_thread(Daemon().stop)
		151	+ # Delayed so the 200 response is sent before the process exits.
		152	+ threading.Timer(0.5, _self_terminate).start()
137	content = {"message": "All engine processes stopped successfully."}	153	content = {"message": "All engine processes stopped successfully."}
138	return Response(status_code=status.HTTP_200_OK, content=json.dumps(content))	154	return Response(status_code=status.HTTP_200_OK, content=json.dumps(content))
139	except Exception as err:	155	except Exception as err:
	@@ -143,6 +159,70 @@ async def stop_instance(request: Request):
143	) from err	159	) from err
144		160
145		161
		162	+@app.post("/node-manager/engine-restart")
		163	+async def engine_restart(request: Request):
		164	+ """Controller-driven engine relaunch without container restart.
		165	+
		166	+ Body: ``{"action": "restart"\|"abort", "instance_id": int?}``
		167	+
		168	+ - ``restart``: kill and re-pull all engine subprocesses in place
		169	+ (``Daemon.restart_engine`` — resolves the launch params, freezes
		170	+ suicide, suspends/resumes the FaultReporter; KV store untouched).
		171	+ Returns 200 once the processes were spawned (model loading continues
		172	+ asynchronously — completion is polled by the Controller via
		173	+ ``/node-manager/status``).
		174	+ - ``abort``: unfreeze the suicide counter — the heartbeat mechanism
		175	+ resumes counting ABNORMAL reports and the pod restarts via k8s
		176	+ (fallback path when engine relaunch failed).
		177	+
		178	+ Forcing this NodeManager to exit (partial-loss coordination) is not an
		179	+ engine-restart concern — the Controller uses ``/node-manager/stop``.
		180	+ """
		181	+ try:
		182	+ payload = await request.json()
		183	+ except Exception as err:
		184	+ raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid JSON body") from err
		185	+ if not isinstance(payload, dict):
		186	+ raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Body must be a JSON object")
		187	+
		188	+ action = payload.get("action")
		189	+ if action not in ("restart", "abort"):
		190	+ raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="'action' must be 'restart' or 'abort'")
		191	+
		192	+ daemon = Daemon()
		193	+ if action == "abort":
		194	+ daemon.unfreeze_suicide()
		195	+ logger.info("Engine restart aborted: suicide arbitration unfrozen (container restart fallback)")
		196	+ return {"message": "abort accepted"}
		197	+
		198	+ # Only reject during an actual snapshot restore: is_started_after_restore
		199	+ # is False in the normal (non-snapshot) deployment, so it alone must
		200	+ # not gate the relaunch.
		201	+ if is_restored_from_host_side_snapshot() and not HeartbeatManager().is_started_after_restore():
		202	+ raise HTTPException(
		203	+ status_code=status.HTTP_409_CONFLICT,
		204	+ detail="Snapshot restore in progress, engine restart not supported",
		205	+ )
		206	+
		207	+ instance_id = payload.get("instance_id")
		208	+ try:
		209	+ await asyncio.to_thread(daemon.restart_engine, instance_id)
		210	+ except EngineRestartInProgressError:
		211	+ raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="Engine restart already in progress")
		212	+ except EngineRestartParamError:
		213	+ raise HTTPException(
		214	+ status_code=status.HTTP_400_BAD_REQUEST, detail="No engine start recorded, nothing to restart"
		215	+ )
		216	+ except Exception as err:
		217	+ logger.error("Failed to restart engines: %s", err)
		218	+ raise HTTPException(
		219	+ status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Engine restart failed: {err}"
		220	+ ) from err
		221	+
		222	+ logger.info("Engines restarted in place for instance %s", instance_id)
		223	+ return {"message": "engine restart accepted"}
		224	+
		225	+
146	@app.post("/node-manager/pause")	226	@app.post("/node-manager/pause")
147	async def pause_instance(request: Request):	227	async def pause_instance(request: Request):
148	"""	228	"""
	@@ -193,13 +273,19 @@ async def _check_node_manager_ready() -> bool:
193		273
194		274
195	@app.get("/node-manager/status")	275	@app.get("/node-manager/status")
196	-async def get_instance_status():	276	+async def get_instance_status(relaxed: bool = False):
197	"""	277	"""
198	Check if all endpoints managed by this node manager are in normal status.	278	Check if all endpoints managed by this node manager are in normal status.
199	- Returns True if all endpoints are normal, False if any endpoint is abnormal.	279	+
		280	+ ``relaxed=true`` (used by the engine-relaunch flow) returns True when no
		281	+ endpoint is ABNORMAL — a freshly relaunched engine reports INITIAL while
		282	+ loading its model, which counts as recovering, not failed.
200	"""	283	"""
201	try:	284	try:
202	- is_normal = await _check_node_manager_ready()	285	+ if relaxed:
		286	+ is_normal = await asyncio.to_thread(HeartbeatManager().check_all_endpoints_recovering)
		287	+ else:
		288	+ is_normal = await _check_node_manager_ready()
203	return {"status": is_normal}	289	return {"status": is_normal}
204	except Exception as err:	290	except Exception as err:
205	logger.error("Failed to check endpoints status: %s", err)	291	logger.error("Failed to check endpoints status: %s", err)
	@@ -275,7 +361,11 @@ class NodeManagerAPI:
275	"""	361	"""
276	return wait_until_api_ready(timeout=timeout)	362	return wait_until_api_ready(timeout=timeout)
277		363
278	- async def stop(self):	364	+ def stop(self):
		365	+ # Synchronous on purpose: Application.stop_all_modules calls
		366	+ # module.stop() without awaiting — an async stop would produce a
		367	+ # coroutine-never-awaited warning on shutdown. The teardown itself
		368	+ # is sync (stop_sync), so the async wrapper was redundant.
279	self.stop_sync()	369	self.stop_sync()
280		370
281	def stop_sync(self):	371	def stop_sync(self):

Mmotor/node_manager/core/daemon.py+388-3

	@@ -9,6 +9,7 @@
9	# See the Mulan PSL v2 for more details.	9	# See the Mulan PSL v2 for more details.
10		10
11	import threading	11	import threading
		12	+import time
12		13
13	from motor.common.resources.instance import PDRole	14	from motor.common.resources.instance import PDRole
14	from motor.common.resources.endpoint import Endpoint	15	from motor.common.resources.endpoint import Endpoint
	@@ -16,6 +17,9 @@ from motor.node_manager.core.services.native_engine.models import RuntimeState
16	from motor.common.utils.singleton import ThreadSafeSingleton	17	from motor.common.utils.singleton import ThreadSafeSingleton
17	from motor.common.logger import get_logger	18	from motor.common.logger import get_logger
18	from motor.config.node_manager import NodeManagerConfig	19	from motor.config.node_manager import NodeManagerConfig
		20	+from motor.node_manager.api_client.controller_api_client import ControllerApiClient
		21	+from motor.node_manager.core.fault_reporter import FaultReporter
		22	+from motor.node_manager.core.heartbeat_manager import HeartbeatManager
19	from motor.node_manager.core.services.protocols import DaemonService, PreparableService	23	from motor.node_manager.core.services.protocols import DaemonService, PreparableService
20	from motor.node_manager.core.services.registry import (	24	from motor.node_manager.core.services.registry import (
21	SERVICE_ENGINE,	25	SERVICE_ENGINE,
	@@ -26,6 +30,25 @@ from motor.node_manager.core.services.registry import (
26	logger = get_logger(__name__)	30	logger = get_logger(__name__)
27		31
28		32
		33	+class EngineRestartInProgressError(RuntimeError):
		34	+ """Raised when a second engine relaunch starts while one is in progress.
		35	+
		36	+ The Controller's dispatch retries may resend a restart command whose
		37	+ response was lost; overlapping relaunches would kill/pull engines twice
		38	+ and collide on ports — the route maps this to 409 and the strategy
		39	+ escalates to container restart.
		40	+ """
		41	+
		42	+
		43	+class EngineRestartParamError(RuntimeError):
		44	+ """Raised when the relaunch cannot resolve its launch params.
		45	+
		46	+ Covers "no engine start recorded" (the NodeManager never received a
		47	+ start command) and "instance id mismatch" (the NodeManager now serves a
		48	+ different instance) — the route maps this to 400.
		49	+ """
		50	+
		51	+
29	class Daemon(ThreadSafeSingleton):	52	class Daemon(ThreadSafeSingleton):
30	"""Orchestrate engine subprocess and KV-store service lifecycle.	53	"""Orchestrate engine subprocess and KV-store service lifecycle.
31		54
	@@ -47,6 +70,9 @@ class Daemon(ThreadSafeSingleton):
47	# Derived from kv_cache_store_config in user_config.json:	70	# Derived from kv_cache_store_config in user_config.json:
48	# mode="combined" (default): engine + KV in same pod	71	# mode="combined" (default): engine + KV in same pod
49	# mode="separated": KV-only pod (no engine, no heartbeat)	72	# mode="separated": KV-only pod (no engine, no heartbeat)
		73	+ self._config = config
		74	+ self.config_lock = threading.RLock()
		75	+
50	# no kv config: engine-only pod	76	# no kv config: engine-only pod
51	kv_cfg = config.kv_cache_store_config	77	kv_cfg = config.kv_cache_store_config
52	if not kv_cfg.enable:	78	if not kv_cfg.enable:
	@@ -73,8 +99,59 @@ class Daemon(ThreadSafeSingleton):
73	self._monitor_stop = threading.Event()	99	self._monitor_stop = threading.Event()
74	self._monitor_interval = 5	100	self._monitor_interval = 5
75		101
		102	+ # --- Suicide arbitration (single decision point for pod rescheduling) ---
		103	+ # The heartbeat module only reports endpoint-state facts; the Daemon
		104	+ # (process/service lifecycle owner) decides whether to kill the pod:
		105	+ # N consecutive abnormal-endpoint observations after the cold-start
		106	+ # grace period, unless the suicide counter is frozen (engine relaunch
		107	+ # window). A deadline (not a boolean) freeze keeps the fallback alive:
		108	+ # if the abort message is lost the freeze expires and counting resumes.
		109	+ self._suicide_abnormal_count = 0
		110	+ self._suicide_freeze_until = 0.0
		111	+ self._should_suicide = False
		112	+ self._suicide_lock = threading.Lock()
		113	+ self._last_endpoints_generation = -1
		114	+ # Consecutive abnormal observations that trigger suicide. Arbitrated
		115	+ # in a dedicated loop paced by the configured heartbeat interval
		116	+ # (matching the former heartbeat cadence, which counted per heartbeat
		117	+ # report): threshold x interval = ~15s of continuous unhealthiness
		118	+ # with the default 3s interval — same as before the move.
		119	+ self._suicide_threshold = 5
		120	+ self._suicide_interval = getattr(config.basic_config, "heartbeat_interval_seconds", 3.0)
		121	+ self._suicide_thread: threading.Thread \| None = None
		122	+ # PIDs whose death was already reported to the Controller (dedup by
		123	+ # pid — a relaunched engine gets a fresh PID, so it can be reported
		124	+ # again on its next death).
		125	+ # Engine relaunch in progress (route serializes via this flag — a
		126	+ # second restart is rejected 409 instead of racing the pull). Owned
		127	+ # here because the Daemon is the process-lifecycle owner; the flag is
		128	+ # set/cleared inside restart_engine, so the lock only guards access.
		129	+ # Engine FT status reporter: the third monitoring source alongside the
		130	+ # process monitor and the suicide arbitration — owned by the Daemon
		131	+ # (the process-lifecycle owner), started when engines are pulled,
		132	+ # paused across an engine relaunch.
		133	+ self._fault_reporter = FaultReporter(config)
		134	+ self._endpoints_info: list[Endpoint] = []
		135	+ # Set once the pulled engines' mgmt ports accept connections — the
		136	+ # HeartbeatManager starts probing engine status only after this
		137	+ # handoff (no engine readiness logic of its own).
		138	+ self._engine_ready = threading.Event()
		139	+ self._engine_restart_in_progress = False
		140	+ self._engine_restart_lock = threading.Lock()
		141	+ self._reported_dead_pids: set[int] = set()
		142	+ # Endpoint ids whose ABNORMAL state was already reported (dedup —
		143	+ # cleared once all endpoints recover, so a re-failure is reported
		144	+ # again).
		145	+ self._reported_abnormal_ep_ids: set[int] = set()
		146	+ # Endpoint ids that have ever been NORMAL. Cold-starting engines
		147	+ # (model loading) are never NORMAL yet — their ABNORMAL observations
		148	+ # must not be reported as engine death (they would trigger a relaunch
		149	+ # that interrupts the loading).
		150	+ self._seen_normal_ep_ids: set[int] = set()
		151	+
76	self._initialized = True	152	self._initialized = True
77	self._start_process_monitor()	153	self._start_process_monitor()
		154	+ self._start_suicide_arbitration()
78		155
79	# ------------------------------------------------------------------	156	# ------------------------------------------------------------------
80	# public API	157	# public API
	@@ -89,6 +166,8 @@ class Daemon(ThreadSafeSingleton):
89	d2d_peer_ips: list[str] \| None = None,	166	d2d_peer_ips: list[str] \| None = None,
90	node_rank: int = 0,	167	node_rank: int = 0,
91	) -> None:	168	) -> None:
		169	+ self._endpoints_info = endpoints_info
		170	+
92	# Phase 1: run PreparableService.prepare() before engines start	171	# Phase 1: run PreparableService.prepare() before engines start
93	for reg in registry.get_preparable():	172	for reg in registry.get_preparable():
94	svc = self._services.get(reg.name)	173	svc = self._services.get(reg.name)
	@@ -107,6 +186,90 @@ class Daemon(ThreadSafeSingleton):
107	node_rank=node_rank,	186	node_rank=node_rank,
108	)	187	)
109		188
		189	+ # Phase 3: signal engine readiness in the background (mgmt ports up),
		190	+ # then start the FT status reporter on the launched engines
		191	+ # (idempotent — a restart keeps the existing thread).
		192	+ self._engine_ready.clear()
		193	+ threading.Thread(
		194	+ target=self._wait_engines_ready,
		195	+ args=(endpoints_info,),
		196	+ daemon=True,
		197	+ name="engine_ready_wait",
		198	+ ).start()
		199	+ self._fault_reporter.start(endpoints_info)
		200	+
		201	+ def restart_engine(self, instance_id: int \| None = None) -> None:
		202	+ """Relaunch the engine service in place, owned entirely by the Daemon.
		203	+
		204	+ Serializes relaunches (second call raises
		205	+ :class:`EngineRestartInProgressError`), resolves the launch params
		206	+ from the RegisterManager (raises :class:`EngineRestartParamError` when
		207	+ nothing was started or the instance id mismatches), freezes suicide
		208	+ arbitration for the relaunch window (unfrozen on failure so the
		209	+ container-restart fallback stays live), suspends the FaultReporter
		210	+ while the engines are down and resumes it afterwards. Used by the
		211	+ Controller-driven engine relaunch flow (``/node-manager/engine-restart``).
		212	+ """
		213	+ with self._engine_restart_lock:
		214	+ if self._engine_restart_in_progress:
		215	+ raise EngineRestartInProgressError()
		216	+ self._engine_restart_in_progress = True
		217	+ try:
		218	+ self._fault_reporter.pause()
		219	+ try:
		220	+ restart_params = self._get_register_manager().get_restart_params()
		221	+ if restart_params is None:
		222	+ raise EngineRestartParamError("no engine start recorded")
		223	+ if instance_id is not None and instance_id != restart_params["instance_id"]:
		224	+ raise EngineRestartParamError("instance id mismatch")
		225	+ with self.config_lock:
		226	+ freeze_sec = self._config.fault_tolerance_config.engine_restart_freeze_sec
		227	+ self.freeze_suicide(freeze_sec)
		228	+ try:
		229	+ # Phase 1: re-run PreparableService.prepare() (idempotent) before engines start
		230	+ for reg in registry.get_preparable():
		231	+ svc = self._services.get(reg.name)
		232	+ if svc is not None and isinstance(svc, PreparableService):
		233	+ svc.prepare(endpoints_count=len(restart_params["endpoints"]))
		234	+
		235	+ # Phase 2: engine service owns its own relaunch lifecycle.
		236	+ engine = self._services.get(SERVICE_ENGINE)
		237	+ if engine is not None:
		238	+ engine.restart( # type: ignore[attr-defined]
		239	+ PDRole(restart_params["role"]),
		240	+ restart_params["endpoints"],
		241	+ restart_params["instance_id"],
		242	+ restart_params["master_dp_ip"],
		243	+ d2d_peer_ips=restart_params["d2d_peer_ips"],
		244	+ node_rank=restart_params["node_rank"],
		245	+ )
		246	+ except Exception:
		247	+ self.unfreeze_suicide()
		248	+ raise
		249	+ finally:
		250	+ self._fault_reporter.resume()
		251	+ finally:
		252	+ with self._engine_restart_lock:
		253	+ self._engine_restart_in_progress = False
		254	+
		255	+ @staticmethod
		256	+ def _get_register_manager():
		257	+ """RegisterManager singleton (owns the engine launch params).
		258	+
		259	+ Lazy import: register_manager pulls in controller_api_client which
		260	+ pulls in more of the module graph; keep it out of daemon import time.
		261	+ """
		262	+ from motor.node_manager.core.register_manager import RegisterManager # pylint: disable=cyclic-import
		263	+
		264	+ return RegisterManager()
		265	+
		266	+ def update_config(self, config: NodeManagerConfig) -> None:
		267	+ """Apply a new config, (re)configuring the FT status reporter."""
		268	+ with self.config_lock:
		269	+ self._config = config
		270	+ self._fault_reporter.update_config(config, self._endpoints_info)
		271	+ logger.info("Daemon configuration updated")
		272	+
110	def pull_kv_store(self) -> None:	273	def pull_kv_store(self) -> None:
111	"""Start/restart the KV store service (if active)."""	274	"""Start/restart the KV store service (if active)."""
112	kv = self._services.get(SERVICE_KV_STORE)	275	kv = self._services.get(SERVICE_KV_STORE)
	@@ -139,11 +302,45 @@ class Daemon(ThreadSafeSingleton):
139	return None	302	return None
140	return engine.metrics_target(endpoint) # type: ignore[attr-defined]	303	return engine.metrics_target(endpoint) # type: ignore[attr-defined]
141		304
		305	+ @property
		306	+ def engine_ready_event(self) -> threading.Event:
		307	+ """Set once the pulled engines' readiness probes succeed.
		308	+
		309	+ Injected into the HeartbeatManager at start time: the status polling
		310	+ waits for this handoff instead of probing engine readiness itself.
		311	+ """
		312	+ return self._engine_ready
		313	+
		314	+ def _wait_engines_ready(self, endpoints_info: list[Endpoint]) -> None:
		315	+ """Background wait for the engines' readiness probes, then flag ready.
		316	+
		317	+ Runs detached so the start command returns immediately; the event is
		318	+ set even when the wait aborted (engine died or the load exceeds the
		319	+ timeout) so the HeartbeatManager never blocks on it — its probing
		320	+ then reports ABNORMAL (or keeps STARTING) and the arbitration handles
		321	+ the death.
		322	+ """
		323	+ engine = self._services.get(SERVICE_ENGINE)
		324	+ if engine is not None:
		325	+ try:
		326	+ engine.wait_ready(endpoints_info) # type: ignore[attr-defined]
		327	+ except Exception:
		328	+ logger.exception("Failed to wait for engines ready")
		329	+ self._engine_ready.set()
		330	+
		331	+ def is_engine_restart_in_progress(self) -> bool:
		332	+ with self._engine_restart_lock:
		333	+ return self._engine_restart_in_progress
		334	+
142	def stop(self) -> None:	335	def stop(self) -> None:
143	self._monitor_stop.set()	336	self._monitor_stop.set()
144	if self._monitor_thread is not None and self._monitor_thread.is_alive():	337	if self._monitor_thread is not None and self._monitor_thread.is_alive():
145	self._monitor_thread.join(timeout=5.0)	338	self._monitor_thread.join(timeout=5.0)
146	self._monitor_thread = None	339	self._monitor_thread = None
		340	+ if self._suicide_thread is not None and self._suicide_thread.is_alive():
		341	+ self._suicide_thread.join(timeout=5.0)
		342	+ self._suicide_thread = None
		343	+ self._fault_reporter.stop()
147		344
148	# Stop services in reverse registration order	345	# Stop services in reverse registration order
149	for svc in reversed(list(self._services.values())):	346	for svc in reversed(list(self._services.values())):
	@@ -172,7 +369,195 @@ class Daemon(ThreadSafeSingleton):
172	while not self._monitor_stop.is_set():	369	while not self._monitor_stop.is_set():
173	for name, svc in self._services.items():	370	for name, svc in self._services.items():
174	try:	371	try:
175	- svc.health_check()	372	+ deaths = svc.health_check() or []
		373	+ self._handle_engine_deaths(deaths)
176	except Exception:	374	except Exception:
177	- logger.exception("health_check failed for service %r", name)	375	+ logger.exception("Health check failed for service %s", name)
178	- self._monitor_stop.wait(self._monitor_interval)	376	+
		377	+ if self._monitor_stop.wait(self._monitor_interval):
		378	+ break
		379	+
		380	+ def _handle_engine_deaths(self, deaths: list) -> None:
		381	+ """Handle engine subprocess deaths: report to Controller, freeze on success.
		382	+
		383	+ The engine-relaunch flow is a base capability — it must work without
		384	+ fault-tolerance reporting enabled, so the Daemon (the process
		385	+ lifecycle owner) detects PID deaths via the monitor's health_check
		386	+ and reports them itself. Dedup by pid: a relaunched engine gets a
		387	+ fresh PID, so the next death is reported again.
		388	+
		389	+ The suicide freeze is only applied after a successful report AND when
		390	+ the in-place relaunch is enabled: a failed report (Controller
		391	+ unreachable) or a disabled relaunch switch means no relaunch is
		392	+ coming, so the arbitration must keep counting and the
		393	+ container-restart fallback (k8s) stays live instead of being frozen.
		394	+ """
		395	+ if not deaths:
		396	+ return
		397	+ for pid, endpoint_id in deaths:
		398	+ if pid in self._reported_dead_pids:
		399	+ continue
		400	+ try:
		401	+ with self.config_lock:
		402	+ pod_ip = self._config.api_config.pod_ip
		403	+ enable_relaunch = self._config.fault_tolerance_config.enable_engine_relaunch
		404	+ freeze_sec = self._config.fault_tolerance_config.engine_restart_wait_timeout_sec
		405	+ if self._report_engine_death(endpoint_id, pod_ip):
		406	+ if enable_relaunch:
		407	+ self.freeze_suicide(freeze_sec)
		408	+ self._reported_dead_pids.add(pid)
		409	+ logger.error(
		410	+ "Engine death reported to Controller: endpoint_id=%s pid=%s (relaunch expected)",
		411	+ endpoint_id,
		412	+ pid,
		413	+ )
		414	+ except Exception as e:
		415	+ logger.error("Failed to handle engine death pid=%s: %s", pid, e)
		416	+
		417	+ @staticmethod
		418	+ def _report_engine_death(endpoint_id: int, pod_ip: str) -> bool:
		419	+ """Report a dead engine to the Controller via the shared software-fault channel."""
		420	+ fault_data = {
		421	+ "exception_type": "EngineDeadError",
		422	+ "exception_message": "Engine process died",
		423	+ "engine_id": endpoint_id,
		424	+ "engine_status": 1,
		425	+ "pod_ip": pod_ip,
		426	+ }
		427	+ try:
		428	+ return ControllerApiClient.report_software_fault(fault_data)
		429	+ except Exception as e:
		430	+ logger.error("Failed to report engine death to Controller: %s", e)
		431	+ return False
		432	+
		433	+ def _start_suicide_arbitration(self) -> None:
		434	+ """Dedicated 3s suicide-arbitration loop.
		435	+
		436	+ Kept separate from the 5s process monitor so the arbitration cadence
		437	+ (3s, matching the former heartbeat rhythm: 5 x 3s ~ 15s to suicide)
		438	+ does not change the process health-check cadence.
		439	+ """
		440	+ if self._suicide_thread is not None and self._suicide_thread.is_alive():
		441	+ return
		442	+ self._suicide_thread = threading.Thread(
		443	+ target=self._suicide_arbitration_loop,
		444	+ daemon=True,
		445	+ name="suicide_arbitration",
		446	+ )
		447	+ self._suicide_thread.start()
		448	+ logger.info("Suicide arbitration thread started (interval=%ss)", self._suicide_interval)
		449	+
		450	+ def _suicide_arbitration_loop(self) -> None:
		451	+ while not self._monitor_stop.is_set():
		452	+ try:
		453	+ self._check_suicide_condition()
		454	+ except Exception:
		455	+ logger.exception("Suicide arbitration failed")
		456	+
		457	+ if self._monitor_stop.wait(self._suicide_interval):
		458	+ break
		459	+
		460	+ # ------------------------------------------------------------------
		461	+ # suicide arbitration (pod rescheduling decision)
		462	+ # ------------------------------------------------------------------
		463	+
		464	+ def _check_suicide_condition(self) -> None:
		465	+ """One arbitration round: observe endpoint health and update the counter.
		466	+
		467	+ State facts come from the HeartbeatManager (endpoint status, grace
		468	+ period, endpoint generation); the decision (consecutive abnormal
		469	+ count >= threshold -> should_suicide) lives here.
		470	+ """
		471	+ hb = HeartbeatManager()
		472	+ generation = hb.endpoints_generation()
		473	+ if generation != self._last_endpoints_generation:
		474	+ # Endpoints were (re)set — restart the counting window.
		475	+ self._last_endpoints_generation = generation
		476	+ with self._suicide_lock:
		477	+ self._suicide_abnormal_count = 0
		478	+ self._should_suicide = False
		479	+ return
		480	+
		481	+ if self.is_suicide_frozen() or hb.is_within_grace_period():
		482	+ with self._suicide_lock:
		483	+ self._suicide_abnormal_count = 0
		484	+ return
		485	+
		486	+ self._seen_normal_ep_ids.update(hb.normal_endpoint_ids())
		487	+ if hb.has_abnormal_endpoints():
		488	+ self._report_abnormal_engines(hb.abnormal_endpoint_ids())
		489	+ with self._suicide_lock:
		490	+ self._suicide_abnormal_count += 1
		491	+ if self._suicide_abnormal_count >= self._suicide_threshold:
		492	+ logger.error(
		493	+ "Reached %d consecutive abnormal endpoint observations, "
		494	+ "setting suicide flag for main to handle (k8s pod restart)",
		495	+ self._suicide_threshold,
		496	+ )
		497	+ self._should_suicide = True
		498	+ else:
		499	+ with self._suicide_lock:
		500	+ self._suicide_abnormal_count = 0
		501	+ # All endpoints recovered — allow re-reporting on the next failure.
		502	+ self._reported_abnormal_ep_ids.clear()
		503	+
		504	+ def _report_abnormal_engines(self, abnormal_ep_ids: list[int]) -> None:
		505	+ """Report ABNORMAL endpoints as dead engines so the Controller relaunches them.
		506	+
		507	+ Covers the case the PID monitor cannot see: the native engine alive
		508	+ but its internal executor (vLLM EngineCore) died — the heartbeat marks
		509	+ the endpoint ABNORMAL (business-level health failed). Dedup per
		510	+ endpoint: a recovered (and re-failed) endpoint is reported again.
		511	+
		512	+ The suicide freeze is only applied after a successful report (see
		513	+ :meth:`_handle_engine_deaths`): a Controller that cannot be reached
		514	+ must not keep the fallback frozen.
		515	+ """
		516	+ for ep_id in abnormal_ep_ids:
		517	+ if ep_id in self._reported_abnormal_ep_ids:
		518	+ continue
		519	+ if ep_id not in self._seen_normal_ep_ids:
		520	+ # Never NORMAL yet = cold start (model loading) — not a death.
		521	+ continue
		522	+ try:
		523	+ with self.config_lock:
		524	+ pod_ip = self._config.api_config.pod_ip
		525	+ enable_relaunch = self._config.fault_tolerance_config.enable_engine_relaunch
		526	+ freeze_sec = self._config.fault_tolerance_config.engine_restart_wait_timeout_sec
		527	+ if self._report_engine_death(ep_id, pod_ip):
		528	+ if enable_relaunch:
		529	+ self.freeze_suicide(freeze_sec)
		530	+ self._reported_abnormal_ep_ids.add(ep_id)
		531	+ logger.error(
		532	+ "Engine death reported to Controller: endpoint_id=%s (ABNORMAL, relaunch expected)",
		533	+ ep_id,
		534	+ )
		535	+ except Exception as e:
		536	+ logger.error("Failed to report abnormal engine endpoint %s: %s", ep_id, e)
		537	+
		538	+ def freeze_suicide(self, seconds: float) -> None:
		539	+ """Freeze suicide arbitration for ``seconds`` and reset counting state.
		540	+
		541	+ Used by the engine relaunch flow: while engines are being re-pulled
		542	+ (or a potential engine fault is being reported to the Controller) the
		543	+ abnormal observations must not accumulate the threshold that would
		544	+ kill this pod before recovery completes.
		545	+ """
		546	+ with self._suicide_lock:
		547	+ self._suicide_freeze_until = time.monotonic() + seconds
		548	+ self._suicide_abnormal_count = 0
		549	+ self._should_suicide = False
		550	+
		551	+ def unfreeze_suicide(self) -> None:
		552	+ """Resume suicide arbitration immediately (container-restart fallback)."""
		553	+ with self._suicide_lock:
		554	+ self._suicide_freeze_until = 0.0
		555	+
		556	+ def is_suicide_frozen(self) -> bool:
		557	+ with self._suicide_lock:
		558	+ return time.monotonic() < self._suicide_freeze_until
		559	+
		560	+ def should_suicide(self) -> bool:
		561	+ """True when the pod should reschedule (main loop handles shutdown)."""
		562	+ with self._suicide_lock:
		563	+ return self._should_suicide

Mmotor/node_manager/core/fault_reporter.py+50-31

	@@ -114,7 +114,13 @@ class FaultReporter:
114	self._enabled = self._compute_enabled(config)	114	self._enabled = self._compute_enabled(config)
115	self._thread: threading.Thread \| None = None	115	self._thread: threading.Thread \| None = None
116	self._stop_event = threading.Event()	116	self._stop_event = threading.Event()
		117	+ self._pause_event = threading.Event()
117	self._endpoints: list[Endpoint] = []	118	self._endpoints: list[Endpoint] = []
		119	+ # Per-endpoint poll state, kept across thread restarts so a restarted
		120	+ # loop does not re-report statuses that were already delivered.
		121	+ self._known_statuses: dict[int, str] = {}
		122	+ self._consecutive_failures: dict[int, int] = {}
		123	+ self._first_poll_time: dict[int, float] = {}
118		124
119	@staticmethod	125	@staticmethod
120	def _compute_enabled(config: NodeManagerConfig) -> bool:	126	def _compute_enabled(config: NodeManagerConfig) -> bool:
	@@ -207,32 +213,52 @@ class FaultReporter:
207	timeout = self._config.fault_tolerance_config.poll_timeout_sec	213	timeout = self._config.fault_tolerance_config.poll_timeout_sec
208	return query_engine_ft_status(ep, timeout)	214	return query_engine_ft_status(ep, timeout)
209		215
		216	+ def pause(self) -> None:
		217	+ """Suspend polling while the engines are being relaunched.
		218	+
		219	+ During a relaunch the engines are killed and re-pulled — their FT
		220	+ endpoints are unreachable, so the poll failures would be reported as
		221	+ deaths while the relaunch is in progress. The loop idles on the pause
		222	+ event and resumes where it left off.
		223	+ """
		224	+ self._pause_event.set()
		225	+ logger.info("FaultReporter paused (engine relaunch in progress)")
		226	+
		227	+ def resume(self) -> None:
		228	+ """Resume polling after the relaunch; reset per-endpoint poll state.
		229	+
		230	+ The engines are fresh processes — the old statuses, poll-failure
		231	+ counts and first-poll timestamps are meaningless. Clearing them
		232	+ restarts the startup-grace window for the re-pulled engines, which
		233	+ makes a separate grace-reset call unnecessary.
		234	+ """
		235	+ self._pause_event.clear()
		236	+ self._known_statuses.clear()
		237	+ self._consecutive_failures.clear()
		238	+ self._first_poll_time.clear()
		239	+ logger.info("FaultReporter resumed after engine relaunch")
		240	+
210	def _main_loop(self) -> None:	241	def _main_loop(self) -> None:
211	"""Poll every engine's FT status and forward faults to Controller."""	242	"""Poll every engine's FT status and forward faults to Controller."""
212	logger.info("FaultReporter loop started.")	243	logger.info("FaultReporter loop started.")
213	- known_statuses: dict[int, str] = {}
214	- consecutive_failures: dict[int, int] = {}
215	- first_poll_time: dict[int, float] = {}
216		244
217	while not self._stop_event.is_set():	245	while not self._stop_event.is_set():
		246	+ if self._pause_event.is_set():
		247	+ if self._stop_event.wait(self._poll_interval_sec()):
		248	+ break
		249	+ continue
218	with self._config_lock:	250	with self._config_lock:
219	endpoints = list(self._endpoints)	251	endpoints = list(self._endpoints)
220	now = time.time()	252	now = time.time()
221	for ep in endpoints:	253	for ep in endpoints:
222	- first_poll_time.setdefault(ep.id, now)	254	+ self._first_poll_time.setdefault(ep.id, now)
223	- self._poll_engine(ep, known_statuses, consecutive_failures, first_poll_time)	255	+ self._poll_engine(ep)
224	if self._stop_event.wait(self._poll_interval_sec()):	256	if self._stop_event.wait(self._poll_interval_sec()):
225	break	257	break
226		258
227	logger.info("FaultReporter loop stopped.")	259	logger.info("FaultReporter loop stopped.")
228		260
229	- def _poll_engine(	261	+ def _poll_engine(self, ep: Endpoint) -> None:
230	- self,
231	- ep: Endpoint,
232	- known_statuses: dict[int, str],
233	- consecutive_failures: dict[int, int],
234	- first_poll_time: dict[int, float],
235	- ) -> None:
236	"""Poll a single engine: forward new non-healthy statuses, or count	262	"""Poll a single engine: forward new non-healthy statuses, or count
237	poll failures and report dead once the threshold is exceeded.	263	poll failures and report dead once the threshold is exceeded.
238	"""	264	"""
	@@ -241,22 +267,22 @@ class FaultReporter:
241	except Exception as e:	267	except Exception as e:
242	# A poll failure must never kill the polling thread — count it	268	# A poll failure must never kill the polling thread — count it
243	# and let the consecutive-failures threshold decide the engine's fate.	269	# and let the consecutive-failures threshold decide the engine's fate.
244	- failures = consecutive_failures.get(ep.id, 0) + 1	270	+ failures = self._consecutive_failures.get(ep.id, 0) + 1
245	- consecutive_failures[ep.id] = failures	271	+ self._consecutive_failures[ep.id] = failures
246	_rl.error_window(	272	_rl.error_window(
247	f"node_manager.fault_reporter.poll.{ep.id}",	273	f"node_manager.fault_reporter.poll.{ep.id}",
248	f"Failed to poll engine {ep.id} FT status: {e}",	274	f"Failed to poll engine {ep.id} FT status: {e}",
249	)	275	)
250	if failures >= self._max_poll_failures():	276	if failures >= self._max_poll_failures():
251	- self._report_unreachable_dead(ep, failures, known_statuses, first_poll_time)	277	+ self._report_unreachable_dead(ep, failures)
252	return	278	return
253		279
254	- consecutive_failures[ep.id] = 0	280	+ self._consecutive_failures[ep.id] = 0
255	try:	281	try:
256	if not isinstance(payload, dict):	282	if not isinstance(payload, dict):
257	raise TypeError(f"unexpected FT status payload type: {type(payload).__name__}")	283	raise TypeError(f"unexpected FT status payload type: {type(payload).__name__}")
258	for engine in payload.get("engines", []):	284	for engine in payload.get("engines", []):
259	- self._process_engine_status(ep.id, engine, known_statuses)	285	+ self._process_engine_status(ep.id, engine)
260	except Exception as e:	286	except Exception as e:
261	# A malformed payload must never kill the polling thread — log and	287	# A malformed payload must never kill the polling thread — log and
262	# continue with the next round.	288	# continue with the next round.
	@@ -269,7 +295,6 @@ class FaultReporter:
269	self,	295	self,
270	ep_id: int,	296	ep_id: int,
271	engine: dict,	297	engine: dict,
272	- known_statuses: dict[int, str],
273	) -> None:	298	) -> None:
274	"""Report a single engine's status if it is non-healthy and new.	299	"""Report a single engine's status if it is non-healthy and new.
275		300
	@@ -285,10 +310,10 @@ class FaultReporter:
285	raise TypeError(f"engine entry of endpoint {ep_id} has no valid status")	310	raise TypeError(f"engine entry of endpoint {ep_id} has no valid status")
286		311
287	if status == ENGINE_STATUS_HEALTHY:	312	if status == ENGINE_STATUS_HEALTHY:
288	- known_statuses[ep_id] = status	313	+ self._known_statuses[ep_id] = status
289	return	314	return
290		315
291	- if known_statuses.get(ep_id) == status:	316	+ if self._known_statuses.get(ep_id) == status:
292	return # already reported	317	return # already reported
293		318
294	engine_status = _ENGINE_STATUS_NAME_TO_INT.get(status)	319	engine_status = _ENGINE_STATUS_NAME_TO_INT.get(status)
	@@ -315,23 +340,17 @@ class FaultReporter:
315	}	340	}
316	# Only mark as reported after successful delivery to Controller	341	# Only mark as reported after successful delivery to Controller
317	if self._send_fault_to_controller(fault_data):	342	if self._send_fault_to_controller(fault_data):
318	- known_statuses[ep_id] = status	343	+ self._known_statuses[ep_id] = status
319		344
320	- def _report_unreachable_dead(	345	+ def _report_unreachable_dead(self, ep: Endpoint, failures: int) -> None:
321	- self,
322	- ep: Endpoint,
323	- failures: int,
324	- known_statuses: dict[int, str],
325	- first_poll_time: dict[int, float],
326	- ) -> None:
327	"""Report an engine as dead after repeated poll failures (deduped).	346	"""Report an engine as dead after repeated poll failures (deduped).
328		347
329	Poll failures during the startup grace period (engine model load)	348	Poll failures during the startup grace period (engine model load)
330	are not reported as dead.	349	are not reported as dead.
331	"""	350	"""
332	- if known_statuses.get(ep.id) == ENGINE_STATUS_DEAD:	351	+ if self._known_statuses.get(ep.id) == ENGINE_STATUS_DEAD:
333	return	352	return
334	- first_poll = first_poll_time.get(ep.id, time.time())	353	+ first_poll = self._first_poll_time.get(ep.id, time.time())
335	if time.time() - first_poll < _STARTUP_GRACE_SEC:	354	if time.time() - first_poll < _STARTUP_GRACE_SEC:
336	logger.debug(	355	logger.debug(
337	"Engine %d unreachable but within startup grace period, not reporting dead",	356	"Engine %d unreachable but within startup grace period, not reporting dead",
	@@ -345,7 +364,7 @@ class FaultReporter:
345	"engine_status": 1,	364	"engine_status": 1,
346	}	365	}
347	if self._send_fault_to_controller(fault_data):	366	if self._send_fault_to_controller(fault_data):
348	- known_statuses[ep.id] = ENGINE_STATUS_DEAD	367	+ self._known_statuses[ep.id] = ENGINE_STATUS_DEAD
349		368
350	def _send_fault_to_controller(self, fault_data: dict) -> bool:	369	def _send_fault_to_controller(self, fault_data: dict) -> bool:
351	"""Inject pod_ip and forward a single fault to Controller.	370	"""Inject pod_ip and forward a single fault to Controller.

Mmotor/node_manager/core/heartbeat_manager.py+106-48

	@@ -20,8 +20,7 @@ from motor.common.utils.singleton import ThreadSafeSingleton
20	from motor.common.utils.snapshot_utils import is_restored_from_host_side_snapshot, RETRY_LOG_FREQUENCY	20	from motor.common.utils.snapshot_utils import is_restored_from_host_side_snapshot, RETRY_LOG_FREQUENCY
21	from motor.config.node_manager import NodeManagerConfig	21	from motor.config.node_manager import NodeManagerConfig
22	from motor.node_manager.api_client.controller_api_client import ControllerApiClient	22	from motor.node_manager.api_client.controller_api_client import ControllerApiClient
23	-from motor.node_manager.core.engine_manager import EngineManager	23	+from motor.node_manager.core.register_manager import RegisterManager
24	-from motor.node_manager.core.daemon import Daemon
25	from motor.node_manager.core.services.native_engine.models import RuntimeState	24	from motor.node_manager.core.services.native_engine.models import RuntimeState
26		25
27		26
	@@ -53,16 +52,12 @@ class HeartbeatManager(ThreadSafeSingleton):
53	daemon=True,	52	daemon=True,
54	name="heartbeat_report",	53	name="heartbeat_report",
55	)	54	)
56	- self._engine_status_thread = threading.Thread(	55	+ # The status thread is created in start() — it needs the Daemon's
57	- target=self._refresh_endpoints_status_loop,	56	+ # engine-ready event injected at that point.
58	- daemon=True,	57	+ self._engine_status_thread: threading.Thread \| None = None
59	- name="endpoint_status_fetch",
60	- )
61	self._thread_started = False	58	self._thread_started = False
62	- self._consecutive_abnormal_count = 0	59	+ self._engine_status_thread_start_time = None
63	- self._abnormal_count_lock = threading.Lock()	60	+ self._is_within_grace_period = True
64	- self._should_suicide = False
65	- self._suicide_lock = threading.Lock()
66	# for snapshot	61	# for snapshot
67	self._register_after_restore_retry_count = 0	62	self._register_after_restore_retry_count = 0
68	self._checkpoint_done_inspect_retry_count = 0	63	self._checkpoint_done_inspect_retry_count = 0
	@@ -74,8 +69,22 @@ class HeartbeatManager(ThreadSafeSingleton):
74	self._initialized = True	69	self._initialized = True
75	logger.info("HeartBeatManager module start.")	70	logger.info("HeartBeatManager module start.")
76		71
77	- def start(self):	72	+ def start(self, engine_ready_event: threading.Event \| None = None) -> None:
		73	+ """Start the heartbeat and engine-status threads.
		74	+
		75	+ ``engine_ready_event`` (the Daemon's engine-ready handoff) gates the
		76	+ status polling: the status thread waits for it before probing the
		77	+ engines, so the HeartbeatManager has no engine-readiness logic of its
		78	+ own. None (snapshot-restore path, no engines pulled) starts probing
		79	+ immediately.
		80	+ """
78	if self._thread_started is False:	81	if self._thread_started is False:
		82	+ self._engine_status_thread = threading.Thread(
		83	+ target=self._refresh_endpoints_status_loop,
		84	+ args=(engine_ready_event,),
		85	+ daemon=True,
		86	+ name="endpoint_status_fetch",
		87	+ )
79	self._heartbeat_report_thread.start()	88	self._heartbeat_report_thread.start()
80	self._engine_status_thread.start()	89	self._engine_status_thread.start()
81	self._thread_started = True	90	self._thread_started = True
	@@ -98,29 +107,76 @@ class HeartbeatManager(ThreadSafeSingleton):
98	for item in node_manager_info.endpoints:	107	for item in node_manager_info.endpoints:
99	self._endpoints.append(item)	108	self._endpoints.append(item)
100	self._endpoints_generation += 1	109	self._endpoints_generation += 1
101	- # Reset abnormal count when endpoints are updated	110	+ self._is_within_grace_period = True
102	- with self._abnormal_count_lock:	111	+ if self._thread_started:
103	- self._consecutive_abnormal_count = 0	112	+ self._engine_status_thread_start_time = time.time()
104	- # Reset suicide flag when endpoints are updated
105	- with self._suicide_lock:
106	- self._should_suicide = False
107		113
108	- def should_suicide(self) -> bool:	114	+ def has_abnormal_endpoints(self) -> bool:
		115	+ """True when any managed endpoint is ABNORMAL.
		116	+
		117	+ State fact for the Daemon's suicide arbitration — the heartbeat
		118	+ module only maintains endpoint state; the Daemon decides whether to
		119	+ kill the pod.
109	"""	120	"""
110	- Check if suicide flag is set.	121	+ with self._endpoint_lock:
111	- Returns True if 5 consecutive abnormal heartbeats have been reported.	122	+ return any(item.status == EndpointStatus.ABNORMAL for item in self._endpoints)
		123	+
		124	+ def normal_endpoint_ids(self) -> list[int]:
		125	+ """Ids of the endpoints currently NORMAL.
		126	+
		127	+ State fact for the Daemon's suicide arbitration: an endpoint that was
		128	+ never NORMAL is still cold-starting (model loading) — its ABNORMAL
		129	+ observations must not be reported as engine death.
112	"""	130	"""
113	- with self._suicide_lock:	131	+ with self._endpoint_lock:
114	- return self._should_suicide	132	+ return [item.id for item in self._endpoints if item.status == EndpointStatus.NORMAL]
		133	+
		134	+ def abnormal_endpoint_ids(self) -> list[int]:
		135	+ """Ids of the endpoints currently ABNORMAL.
		136	+
		137	+ ABNORMAL covers both native-engine process death and business-level
		138	+ failure (engine alive but its internal executor died — vLLM EngineCore
		139	+ crash) — either way the engine is unusable and must be relaunched.
		140	+ """
		141	+ with self._endpoint_lock:
		142	+ return [item.id for item in self._endpoints if item.status == EndpointStatus.ABNORMAL]
		143	+
		144	+ def endpoints_generation(self) -> int:
		145	+ """Incremented on every endpoint update (Daemon resets its suicide
		146	+ count when this changes).
		147	+ """
		148	+ with self._endpoint_lock:
		149	+ return self._endpoints_generation
		150	+
		151	+ def is_within_grace_period(self) -> bool:
		152	+ """True during the 120s cold-start window after endpoints were set."""
		153	+ with self._endpoint_lock:
		154	+ return self._is_within_grace_period
115		155
116	def stop(self) -> None:	156	def stop(self) -> None:
117	self.stop_event.set()	157	self.stop_event.set()
118	if self._heartbeat_report_thread.is_alive():	158	if self._heartbeat_report_thread.is_alive():
119	self._heartbeat_report_thread.join(timeout=2.0)	159	self._heartbeat_report_thread.join(timeout=2.0)
120	- if self._engine_status_thread.is_alive():	160	+ if self._engine_status_thread is not None and self._engine_status_thread.is_alive():
121	self._engine_status_thread.join(timeout=2.0)	161	self._engine_status_thread.join(timeout=2.0)
122	logger.info("HeartBeatManager stopped.")	162	logger.info("HeartBeatManager stopped.")
123		163
		164	+ def check_all_endpoints_recovering(self) -> bool:
		165	+ """True when no managed endpoint is ABNORMAL.
		166	+
		167	+ Relaxed readiness for the engine-relaunch flow: an engine that was
		168	+ just relaunched reports INITIAL while loading its model — the relaunch
		169	+ succeeded as soon as every endpoint is past ABNORMAL (engine process
		170	+ alive); model readiness is tracked separately by NORMAL.
		171	+ """
		172	+ with self._endpoint_lock:
		173	+ if not self._endpoints:
		174	+ return False
		175	+ for endpoint in self._endpoints:
		176	+ if endpoint.status == EndpointStatus.ABNORMAL:
		177	+ return False
		178	+ return True
		179	+
124	def check_all_endpoints_normal(self) -> bool:	180	def check_all_endpoints_normal(self) -> bool:
125	"""	181	"""
126	Check if all endpoints are in normal status.	182	Check if all endpoints are in normal status.
	@@ -160,6 +216,10 @@ class HeartbeatManager(ThreadSafeSingleton):
160		216
161	def get_engine_metrics_targets(self) -> list[str]:	217	def get_engine_metrics_targets(self) -> list[str]:
162	"""Return native metrics URLs for routable local endpoints."""	218	"""Return native metrics URLs for routable local endpoints."""
		219	+ # Lazy import: the Daemon imports this module at top level; this
		220	+ # runtime lookup keeps the module graph acyclic.
		221	+ from motor.node_manager.core.daemon import Daemon # pylint: disable=cyclic-import
		222	+
163	with self._endpoint_lock:	223	with self._endpoint_lock:
164	endpoints = [endpoint for endpoint in self._endpoints if not endpoint.headless]	224	endpoints = [endpoint for endpoint in self._endpoints if not endpoint.headless]
165	daemon = Daemon()	225	daemon = Daemon()
	@@ -185,7 +245,17 @@ class HeartbeatManager(ThreadSafeSingleton):
185	with self._started_after_restore_lock:	245	with self._started_after_restore_lock:
186	self._is_started_after_restore = is_started	246	self._is_started_after_restore = is_started
187		247
188	- def _refresh_endpoints_status_loop(self) -> None:	248	+ def _refresh_endpoints_status_loop(self, engine_ready_event: threading.Event \| None = None) -> None:
		249	+ # Wait for the Daemon's engine-ready handoff (mgmt ports up) before
		250	+ # probing — engine readiness is the Daemon's/engine service's concern,
		251	+ # not this module's. The event is always set eventually (also when the
		252	+ # engines died), so this never blocks indefinitely.
		253	+ if engine_ready_event is not None:
		254	+ while not self.stop_event.is_set():
		255	+ if engine_ready_event.wait(timeout=1.0):
		256	+ break
		257	+ if self.stop_event.is_set():
		258	+ return
189	while not self.stop_event.is_set():	259	while not self.stop_event.is_set():
190	self._refresh_native_engine_status()	260	self._refresh_native_engine_status()
191	self.stop_event.wait(1)	261	self.stop_event.wait(1)
	@@ -198,6 +268,10 @@ class HeartbeatManager(ThreadSafeSingleton):
198	if not endpoints_snapshot:	268	if not endpoints_snapshot:
199	return	269	return
200		270
		271	+ # Lazy import: the Daemon imports this module at top level; this
		272	+ # runtime lookup keeps the module graph acyclic.
		273	+ from motor.node_manager.core.daemon import Daemon # pylint: disable=cyclic-import
		274	+
201	updated_endpoints = []	275	updated_endpoints = []
202	daemon = Daemon()	276	daemon = Daemon()
203	for item in endpoints_snapshot:	277	for item in endpoints_snapshot:
	@@ -229,7 +303,7 @@ class HeartbeatManager(ThreadSafeSingleton):
229	item.status = original_status	303	item.status = original_status
230	elif runtime_state in (RuntimeState.STARTING, RuntimeState.STOPPING):	304	elif runtime_state in (RuntimeState.STARTING, RuntimeState.STOPPING):
231	# Loading is not a failure. Keep INITIAL (or the last reported	305	# Loading is not a failure. Keep INITIAL (or the last reported
232	- # status) until the native readiness endpoint succeeds.	306	+ # status) until the native readiness probe succeeds.
233	logger.debug(	307	logger.debug(
234	"Native engine %s is %s, keeping status %s",	308	"Native engine %s is %s, keeping status %s",
235	format_address(item.ip, item.business_port),	309	format_address(item.ip, item.business_port),
	@@ -260,12 +334,9 @@ class HeartbeatManager(ThreadSafeSingleton):
260		334
261	def _report_heartbeat_loop(self) -> None:	335	def _report_heartbeat_loop(self) -> None:
262	while not self.stop_event.is_set():	336	while not self.stop_event.is_set():
263	- has_abnormal = False
264	is_normal = True	337	is_normal = True
265	try:	338	try:
266	with self._endpoint_lock:	339	with self._endpoint_lock:
267	- # Check actual endpoint status, not the reported status.
268	- has_abnormal = any(item.status == EndpointStatus.ABNORMAL for item in self._endpoints)
269	is_normal = all(item.status == EndpointStatus.NORMAL for item in self._endpoints)	340	is_normal = all(item.status == EndpointStatus.NORMAL for item in self._endpoints)
270		341
271	endpoint_status_list = {item.id: item.status for item in self._endpoints}	342	endpoint_status_list = {item.id: item.status for item in self._endpoints}
	@@ -275,7 +346,7 @@ class HeartbeatManager(ThreadSafeSingleton):
275	if (	346	if (
276	is_normal	347	is_normal
277	and not is_restored_from_host_side_snapshot()	348	and not is_restored_from_host_side_snapshot()
278	- and not EngineManager().is_engine_checkpoint_done()	349	+ and not RegisterManager().is_engine_checkpoint_done()
279	):	350	):
280	if self._checkpoint_done_inspect_retry_count % RETRY_LOG_FREQUENCY == 0:	351	if self._checkpoint_done_inspect_retry_count % RETRY_LOG_FREQUENCY == 0:
281	logger.info(	352	logger.info(
	@@ -317,28 +388,15 @@ class HeartbeatManager(ThreadSafeSingleton):
317	window_sec=60,	388	window_sec=60,
318	)	389	)
319		390
320	- # Update consecutive abnormal count after successful heartbeat report	391	+ # Suicide arbitration moved to the Daemon's process monitor:
321	- with self._abnormal_count_lock:	392	+ # this module only reports the endpoint-state facts.
322	- if has_abnormal:
323	- self._consecutive_abnormal_count += 1
324	- logger.warning("Consecutive abnormal heartbeat count: %d/5", self._consecutive_abnormal_count)
325	- # Set suicide flag if reached 5 consecutive abnormal heartbeats
326	- if self._consecutive_abnormal_count >= 5:
327	- logger.error(
328	- "Reached 5 consecutive abnormal heartbeats, setting suicide flag for main to handle..."
329	- )
330	- with self._suicide_lock:
331	- self._should_suicide = True
332	- else:
333	- self._consecutive_abnormal_count = 0
334	-
335	with self.config_lock:	393	with self.config_lock:
336	time.sleep(self.heartbeat_interval_seconds)	394	time.sleep(self.heartbeat_interval_seconds)
337		395
338	def _register_after_restore(self) -> None:	396	def _register_after_restore(self) -> None:
339	# refresh config: job_name from snapshot metadata and new pod ip	397	# refresh config: job_name from snapshot metadata and new pod ip
340	try:	398	try:
341	- EngineManager().register_prepare_after_restore()	399	+ RegisterManager().register_prepare_after_restore()
342	except Exception as e:	400	except Exception as e:
343	if self._register_after_restore_retry_count % RETRY_LOG_FREQUENCY == 0:	401	if self._register_after_restore_retry_count % RETRY_LOG_FREQUENCY == 0:
344	logger.error("[snapshot] Failed to register prepare after restore: %s", e)	402	logger.error("[snapshot] Failed to register prepare after restore: %s", e)
	@@ -348,11 +406,11 @@ class HeartbeatManager(ThreadSafeSingleton):
348	# Register for post-snapshot brandnew job name	406	# Register for post-snapshot brandnew job name
349	# Do not consider retry	407	# Do not consider retry
350	# If current register failed, next register will be triggered by next heartbeat report exception	408	# If current register failed, next register will be triggered by next heartbeat report exception
351	- ret = EngineManager().post_register_msg()	409	+ ret = RegisterManager().post_register_msg()
352	self._is_registered_after_restore = ret is True	410	self._is_registered_after_restore = ret is True
353		411
354	def _reregister(self) -> None:	412	def _reregister(self) -> None:
355	- ret = EngineManager().post_reregister_msg()	413	+ ret = RegisterManager().post_reregister_msg()
356	if ret is False:	414	if ret is False:
357	logger.error("reregister failed")	415	logger.error("reregister failed")
358	else:	416	else:

Rmotor/node_manager/core/engine_manager.py→motor/node_manager/core/register_manager.py+27-18

	@@ -31,13 +31,12 @@ from motor.common.utils.snapshot_utils import (
31	)	31	)
32	from motor.config.node_manager import HardwareType, NodeManagerConfig	32	from motor.config.node_manager import HardwareType, NodeManagerConfig
33	from motor.node_manager.api_client.controller_api_client import ControllerApiClient	33	from motor.node_manager.api_client.controller_api_client import ControllerApiClient
34	-from motor.node_manager.core.fault_reporter import FaultReporter
35	from motor.node_manager.core.api_ready_event import wait_until_api_ready	34	from motor.node_manager.core.api_ready_event import wait_until_api_ready
36		35
37	logger = get_logger(__name__)	36	logger = get_logger(__name__)
38		37
39		38
40	-class EngineManager(ThreadSafeSingleton):	39	+class RegisterManager(ThreadSafeSingleton):
41	def __init__(self, config: NodeManagerConfig \| None = None) -> None:	40	def __init__(self, config: NodeManagerConfig \| None = None) -> None:
42	if hasattr(self, "_initialized"):	41	if hasattr(self, "_initialized"):
43	return	42	return
	@@ -52,35 +51,26 @@ class EngineManager(ThreadSafeSingleton):
52	self.instance_id: int = 0	51	self.instance_id: int = 0
53	self.d2d_peer_ips: list[str] \| None = None	52	self.d2d_peer_ips: list[str] \| None = None
54	self.node_rank: int = 0	53	self.node_rank: int = 0
		54	+ # Persisted from the start command for later engine relaunch (the
		55	+ # Controller does not re-send StartCmdMsg on relaunch).
		56	+ self.master_dp_ip: str = ""
		57	+ self.role: str = ""
55	self.is_working = False	58	self.is_working = False
56		59
57	# for snapshot restore, should be recorded during a snapshot-enabled cold start	60	# for snapshot restore, should be recorded during a snapshot-enabled cold start
58	self.is_snapshot_master = False	61	self.is_snapshot_master = False
59		62
60	- self._fault_reporter = FaultReporter(config)
61	-
62	self._register_thread = threading.Thread(target=self._register, daemon=True, name="engine_register")	63	self._register_thread = threading.Thread(target=self._register, daemon=True, name="engine_register")
63	self._register_thread.start()	64	self._register_thread.start()
64		65
65	self._initialized = True	66	self._initialized = True
66	logger.info("Engine Manager module initialized.")	67	logger.info("Engine Manager module initialized.")
67		68
68	- def start(self) -> None:
69	- """Start engine manager background threads."""
70	- self._fault_reporter.start(self.endpoints)
71	- logger.info("EngineManager started.")
72	-
73	def update_config(self, config: NodeManagerConfig) -> None:	69	def update_config(self, config: NodeManagerConfig) -> None:
74	- """Update configuration for the engine manager.	70	+ """Update configuration for the engine manager."""
75	-
76	- Supports dynamically enabling/disabling the fault reporting thread
77	- when enable_fault_tolerance changes.
78	- """
79	with self.config_lock:	71	with self.config_lock:
80	self._config = config	72	self._config = config
81	-	73	+ logger.info("RegisterManager configuration updated.")
82	- self._fault_reporter.update_config(config, self.endpoints)
83	- logger.info("EngineManager configuration updated.")
84		74
85	def get_snapshot_metadata_path(self) -> str:	75	def get_snapshot_metadata_path(self) -> str:
86	# if snapshot_metadata_path is set, return it, otherwise using configmap mounted snapshot_metadata.json and return default MOTOR_SNAPSHOT_METADATA_PATH	76	# if snapshot_metadata_path is set, return it, otherwise using configmap mounted snapshot_metadata.json and return default MOTOR_SNAPSHOT_METADATA_PATH
	@@ -228,6 +218,8 @@ class EngineManager(ThreadSafeSingleton):
228	self.endpoints = start_cmd.endpoints	218	self.endpoints = start_cmd.endpoints
229	self.d2d_peer_ips = start_cmd.d2d_peer_ips	219	self.d2d_peer_ips = start_cmd.d2d_peer_ips
230	self.node_rank = start_cmd.node_rank	220	self.node_rank = start_cmd.node_rank
		221	+ self.master_dp_ip = start_cmd.master_dp_ip
		222	+ self.role = start_cmd.role
231		223
232	if (	224	if (
233	self._config.snapshot_config.enable_snapshot	225	self._config.snapshot_config.enable_snapshot
	@@ -239,8 +231,25 @@ class EngineManager(ThreadSafeSingleton):
239	self._write_ranktable_to_file(start_cmd.ranktable)	231	self._write_ranktable_to_file(start_cmd.ranktable)
240	return True	232	return True
241		233
		234	+ def get_restart_params(self) -> dict \| None:
		235	+ """Snapshot of the parameters needed to relaunch the engines.
		236	+
		237	+ Returns None when no start command was received yet (nothing to
		238	+ relaunch); the Controller does not re-send StartCmdMsg on relaunch,
		239	+ so the Daemon rebuilds the pull arguments from this snapshot.
		240	+ """
		241	+ if self.instance_id <= 0 or not self.endpoints or not self.role:
		242	+ return None
		243	+ return {
		244	+ "role": self.role,
		245	+ "endpoints": list(self.endpoints),
		246	+ "instance_id": self.instance_id,
		247	+ "master_dp_ip": self.master_dp_ip,
		248	+ "d2d_peer_ips": self.d2d_peer_ips,
		249	+ "node_rank": self.node_rank,
		250	+ }
		251	+
242	def stop(self) -> None:	252	def stop(self) -> None:
243	- self._fault_reporter.stop()
244	try:	253	try:
245	if hasattr(self, "_register_thread") and self._register_thread.is_alive():	254	if hasattr(self, "_register_thread") and self._register_thread.is_alive():
246	self._register_thread.join(timeout=2.0)	255	self._register_thread.join(timeout=2.0)

Mmotor/node_manager/core/services/native_engine/backends/base.py+3-0

	@@ -10,6 +10,7 @@
10		10
11	from typing import Protocol	11	from typing import Protocol
12		12
		13	+from motor.common.logger import get_logger
13	from motor.config.endpoint import EndpointConfig	14	from motor.config.endpoint import EndpointConfig
14	from motor.node_manager.core.services.native_engine.models import (	15	from motor.node_manager.core.services.native_engine.models import (
15	CommandSpec,	16	CommandSpec,
	@@ -18,6 +19,8 @@ from motor.node_manager.core.services.native_engine.models import (
18	ProbeSpec,	19	ProbeSpec,
19	)	20	)
20		21
		22	+logger = get_logger(__name__)
		23	+
21		24
22	class NativeEngineBackend(Protocol):	25	class NativeEngineBackend(Protocol):
23	"""Stateless conversion from launch context to a native engine launch specification."""	26	"""Stateless conversion from launch context to a native engine launch specification."""

Mmotor/node_manager/core/services/native_engine/service.py+80-22

	@@ -9,12 +9,11 @@
9	# See the Mulan PSL v2 for more details.	9	# See the Mulan PSL v2 for more details.
10		10
11	import os	11	import os
12	-import signal
13	import threading	12	import threading
		13	+import time
14		14
15	from motor.common.resources.endpoint import Endpoint	15	from motor.common.resources.endpoint import Endpoint
16	from motor.common.resources.instance import PDRole	16	from motor.common.resources.instance import PDRole
17	-from motor.common.utils.env import Env
18	from motor.common.logger import get_logger	17	from motor.common.logger import get_logger
19	from motor.common.utils.net import format_address	18	from motor.common.utils.net import format_address
20	from motor.node_manager.core.services.native_engine.factory import get_backend	19	from motor.node_manager.core.services.native_engine.factory import get_backend
	@@ -72,9 +71,9 @@ class NativeEngineService:
72	self.backend = get_backend(self.engine_type)	71	self.backend = get_backend(self.engine_type)
73	self.supervisor = ProcessSupervisor()	72	self.supervisor = ProcessSupervisor()
74	self._pull_lock = threading.Lock()	73	self._pull_lock = threading.Lock()
75	-	74	+ # Number of engine relaunches performed in this container's lifetime;
76	- self.restart_on_failure = Env.motor_restart_engine	75	+ # used to label the log separators between successive engine launches.
77	- self._recovery_requested = False	76	+ self._restart_count = 0
78		77
79	def pull(	78	def pull(
80	self,	79	self,
	@@ -141,12 +140,10 @@ class NativeEngineService:
141	if self.supervisor.start(endpoint.id, launch_spec.command, launch_spec.probe):	140	if self.supervisor.start(endpoint.id, launch_spec.command, launch_spec.probe):
142	started_endpoint_ids.append(endpoint.id)	141	started_endpoint_ids.append(endpoint.id)
143		142
144	- self._recovery_requested = False
145	-
146	except Exception as e:	143	except Exception as e:
147	for endpoint_id in reversed(started_endpoint_ids):	144	for endpoint_id in reversed(started_endpoint_ids):
148	self.supervisor.stop(endpoint_id)	145	self.supervisor.stop(endpoint_id)
149	- raise RuntimeError("Failed to pull engine: %s" % e) from e	146	+ raise RuntimeError(f"Failed to pull engine: {e}") from e
150		147
151	def stop(self) -> list[int]:	148	def stop(self) -> list[int]:
152	"""Gracefully stop all native process groups, then force-kill on timeout."""	149	"""Gracefully stop all native process groups, then force-kill on timeout."""
	@@ -168,27 +165,88 @@ class NativeEngineService:
168	scheme = "https" if probe.tls_config and probe.tls_config.enable_tls else "http"	165	scheme = "https" if probe.tls_config and probe.tls_config.enable_tls else "http"
169	return f"{scheme}://{format_address(endpoint.ip, endpoint.business_port)}/metrics"	166	return f"{scheme}://{format_address(endpoint.ip, endpoint.business_port)}/metrics"
170		167
171	- def health_check(self) -> None:	168	+ def restart(
172	- """Trigger Pod-level recovery when any native engine process exits."""	169	+ self,
173	- dead_pids = self.supervisor.dead_pids()	170	+ pd_role_info: PDRole,
174	- if not dead_pids:	171	+ endpoints_info: list[Endpoint],
175	- return	172	+ instance_id: int,
176	- logger.warning(	173	+ master_dp_ip: str,
177	- "Engine PIDs %s died (restart_on_failure=%s)",	174	+ d2d_peer_ips: list[str] \| None = None,
178	- dead_pids,	175	+ node_rank: int = 0,
179	- self.restart_on_failure,	176	+ ) -> None:
		177	+ """Relaunch the native engines in place: stop the old process groups,
		178	+ then pull fresh ones. Owns the whole relaunch lifecycle (the Daemon
		179	+ stays engine-agnostic).
		180	+
		181	+ The engines inherit this process's stdout/stderr, so every relaunch
		182	+ appends to the same container log file — a prominent separator is
		183	+ printed between the old and the new engine logs to mark the relaunch
		184	+ number.
		185	+ """
		186	+ logger.info("Restarting native engines for instance %d (stop -> pull)", instance_id)
		187	+ self.stop()
		188	+ self._log_restart_separator(instance_id)
		189	+ self.pull(pd_role_info, endpoints_info, instance_id, master_dp_ip, d2d_peer_ips, node_rank)
		190	+
		191	+ def _log_restart_separator(self, instance_id: int) -> None:
		192	+ """Print a prominent separator marking the N-th engine relaunch.
		193	+
		194	+ Engine subprocesses inherit this process's stdout/stderr, so their
		195	+ logs accumulate in the same container log file. This banner (printed
		196	+ between the old engines' stop and the new engines' pull) makes each
		197	+ relaunch clearly delimited and greppable: ``[ENGINE RELAUNCH #N]``.
		198	+ """
		199	+ self._restart_count += 1
		200	+ banner = (
		201	+ "\n"
		202	+ f"{'=' * 24} [ENGINE RELAUNCH #{self._restart_count}] "
		203	+ f"instance_id={instance_id} {time.strftime('%Y-%m-%d %H:%M:%S')} "
		204	+ f"{'=' * 24}\n"
180	)	205	)
181	- if self.restart_on_failure and not self._recovery_requested:	206	+ # Straight to the container stdout stream (k8s collects it as the
182	- self._recovery_requested = True	207	+ # container log file) — independent of this process's logger setup.
183	- logger.info("Engine failure requires Pod-level recovery")	208	+ print(banner, flush=True)
184	- os.kill(os.getpid(), signal.SIGTERM)	209	+ logger.info("Engine relaunch #%d separator printed to container log", self._restart_count)
		210	+
		211	+ def wait_ready(self, endpoints: list[Endpoint], timeout: float = 60.0) -> None:
		212	+ """Wait until every native engine's readiness probe succeeds.
		213	+
		214	+ The probe hits the engine's business port (the OpenAI API), which only
		215	+ accepts connections once the model finished loading — so this waits
		216	+ out the (potentially long) model load. Returns on timeout as well:
		217	+ the HeartbeatManager's STARTING-preserves-status semantics keep the
		218	+ loading window from being misreported as a death.
		219	+ """
		220	+ deadline = time.monotonic() + timeout
		221	+ for endpoint in endpoints:
		222	+ address = format_address(endpoint.ip, endpoint.business_port)
		223	+ logger.info("Waiting for native engine at %s to become ready...", address)
		224	+ while time.monotonic() < deadline:
		225	+ if self.supervisor.state(endpoint.id, endpoint.ip, int(endpoint.business_port)) == RuntimeState.READY:
		226	+ logger.info("Native engine at %s is ready.", address)
		227	+ break
		228	+ time.sleep(1)
		229	+
		230	+ def health_check(self) -> list:
		231	+ """Return deaths ``[(pid, endpoint_id)]`` for the Daemon's death handling.
		232	+
		233	+ Whether to freeze suicide arbitration and wait for an in-place
		234	+ relaunch, or let the pod self-terminate (k8s restarts the container),
		235	+ is the Daemon's decision — gated by ``enable_engine_relaunch`` in the
		236	+ NodeManager config. This service only surfaces dead PIDs.
		237	+ """
		238	+ dead = self.supervisor.dead_pids()
		239	+ if not dead:
		240	+ return []
		241	+ logger.warning("Engine PIDs %s died", [pid for pid, _ in dead])
		242	+ return dead
185		243
186	def _calc_visible_device_ids(self, index: int, device_size: int) -> str:	244	def _calc_visible_device_ids(self, index: int, device_size: int) -> str:
187	local_world_size = self.parallel_config.local_world_size	245	local_world_size = self.parallel_config.local_world_size
188	start_device_id = index * local_world_size % device_size	246	start_device_id = index * local_world_size % device_size
189	end_device_id = start_device_id + local_world_size	247	end_device_id = start_device_id + local_world_size
190	if end_device_id > device_size:	248	if end_device_id > device_size:
191	- device_ids = list(range(start_device_id, device_size)) + list(range(0, end_device_id - device_size))	249	+ device_ids = list(range(start_device_id, device_size)) + list(range(end_device_id - device_size))
192	else:	250	else:
193	device_ids = list(range(start_device_id, end_device_id))	251	device_ids = list(range(start_device_id, end_device_id))
194	if self.single_container_flag:	252	if self.single_container_flag:

Mmotor/node_manager/core/services/native_engine/supervisor.py+8-2

	@@ -118,7 +118,13 @@ class ProcessSupervisor:
118	logger.warning("Native health probe failed for endpoint %s: %s", endpoint_id, err)	118	logger.warning("Native health probe failed for endpoint %s: %s", endpoint_id, err)
119	return self._commit_state(endpoint_id, runtime, RuntimeState.UNHEALTHY)	119	return self._commit_state(endpoint_id, runtime, RuntimeState.UNHEALTHY)
120		120
121	- def dead_pids(self) -> list[int]:	121	+ def dead_pids(self) -> list[tuple[int, int]]:
		122	+ """Return ``(pid, endpoint_id)`` for processes that died.
		123	+
		124	+ The endpoint id is included so the caller can report the death per
		125	+ endpoint (the Daemon's engine-relaunch flow dedups and reports by
		126	+ pid/endpoint).
		127	+ """
122	with self._lock:	128	with self._lock:
123	dead = []	129	dead = []
124	dead_runtimes = []	130	dead_runtimes = []
	@@ -126,7 +132,7 @@ class ProcessSupervisor:
126	if runtime.state == RuntimeState.STOPPING or runtime.process.poll() is None:	132	if runtime.state == RuntimeState.STOPPING or runtime.process.poll() is None:
127	continue	133	continue
128	runtime.state = RuntimeState.STOPPED	134	runtime.state = RuntimeState.STOPPED
129	- dead.append(runtime.process.pid)	135	+ dead.append((runtime.process.pid, endpoint_id))
130	dead_runtimes.append(runtime)	136	dead_runtimes.append(runtime)
131	self._processes.pop(endpoint_id, None)	137	self._processes.pop(endpoint_id, None)
132		138

Mmotor/node_manager/core/services/protocols.py+5-1

	@@ -26,12 +26,16 @@ class DaemonService(Protocol):
26	"""Stop the service. Must be idempotent."""	26	"""Stop the service. Must be idempotent."""
27	...	27	...
28		28
29	- def health_check(self) -> None:	29	+ def health_check(self) -> list \| None:
30	"""Check service health and self-restart if needed.	30	"""Check service health and self-restart if needed.
31		31
32	Called by the Daemon process monitor on every tick (~5 s).	32	Called by the Daemon process monitor on every tick (~5 s).
33	The implementation handles its own failure detection and recovery	33	The implementation handles its own failure detection and recovery
34	(e.g. ``os.kill`` for subprocess PIDs, ``thread.is_alive`` for threads).	34	(e.g. ``os.kill`` for subprocess PIDs, ``thread.is_alive`` for threads).
		35	+
		36	+ Returns a list of death events ``[(pid, endpoint_id), ...]`` for
		37	+ services with subprocesses (the engine service); other services may
		38	+ return None.
35	"""	39	"""
36	...	40	...
37		41

Mmotor/node_manager/node_manager.py+8-8

	@@ -16,7 +16,7 @@ from motor.common.utils.env import Env
16	from motor.config.node_manager import NodeManagerConfig	16	from motor.config.node_manager import NodeManagerConfig
17	from motor.node_manager.api_server.node_manager_api import NodeManagerAPI	17	from motor.node_manager.api_server.node_manager_api import NodeManagerAPI
18	from motor.node_manager.core.daemon import Daemon	18	from motor.node_manager.core.daemon import Daemon
19	-from motor.node_manager.core.engine_manager import EngineManager	19	+from motor.node_manager.core.register_manager import RegisterManager
20	from motor.node_manager.core.heartbeat_manager import HeartbeatManager	20	from motor.node_manager.core.heartbeat_manager import HeartbeatManager
21		21
22	logger = get_logger(__name__)	22	logger = get_logger(__name__)
	@@ -26,7 +26,7 @@ class NodeManager(Application):
26	"""Orchestrates NodeManager modules and daemon lifecycle.	26	"""Orchestrates NodeManager modules and daemon lifecycle.
27		27
28	Module set depends on the active service profile:	28	Module set depends on the active service profile:
29	- * engine active → EngineManager + HeartbeatManager (registration, heartbeats)	29	+ * engine active → RegisterManager + HeartbeatManager (registration, heartbeats)
30	* engine absent → only Daemon + NodeManagerAPI (KV-only pod)	30	* engine absent → only Daemon + NodeManagerAPI (KV-only pod)
31	"""	31	"""
32		32
	@@ -48,7 +48,7 @@ class NodeManager(Application):
48	self.add_module("NodeManagerAPI", NodeManagerAPI(config=self.config))	48	self.add_module("NodeManagerAPI", NodeManagerAPI(config=self.config))
49		49
50	if daemon.has_engine:	50	if daemon.has_engine:
51	- self.add_module("EngineManager", EngineManager(self.config))	51	+ self.add_module("RegisterManager", RegisterManager(self.config))
52	self.add_module("HeartbeatManager", HeartbeatManager(self.config))	52	self.add_module("HeartbeatManager", HeartbeatManager(self.config))
53		53
54	logger.info(	54	logger.info(
	@@ -81,7 +81,7 @@ class NodeManager(Application):
81		81
82	def _on_daemon_tick(self) -> None:	82	def _on_daemon_tick(self) -> None:
83	if self._check_suicide():	83	if self._check_suicide():
84	- logger.error("Detected suicide flag from HeartbeatManager")	84	+ logger.error("Detected suicide flag from Daemon (k8s pod rescheduling)")
85	self.stop_event.set()	85	self.stop_event.set()
86		86
87	def shutdown(self) -> None:	87	def shutdown(self) -> None:
	@@ -94,8 +94,8 @@ class NodeManager(Application):
94	# ------------------------------------------------------------------	94	# ------------------------------------------------------------------
95		95
96	def _check_suicide(self) -> bool:	96	def _check_suicide(self) -> bool:
97	- """True when HeartbeatManager requests pod rescheduling."""	97	+ """True when the Daemon (suicide arbitrator) requests pod rescheduling."""
98	- hb = self.get_module("HeartbeatManager")	98	+ daemon = self.get_module("Daemon")
99	- if hb is None:	99	+ if daemon is None:
100	return False	100	return False
101	- return hb.should_suicide()	101	+ return daemon.should_suicide()

Atests/controller/api_client/test_node_manager_api_client.py+58-0

@@ -0,0 +1,58 @@
	1	+# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
	2	+# MindIE is licensed under Mulan PSL v2.
	3	+# You can use this software according to the terms and conditions of the Mulan PSL v2.
	4	+# You may obtain a copy of Mulan PSL v2 at:
	5	+# http://license.coscl.org.cn/MulanPSL2
	6	+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
	7	+# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
	8	+# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
	9	+# See the Mulan PSL v2 for more details.
	10	+"""Tests for the Controller -> NodeManager client (engine restart dispatch)."""
	11	+
	12	+from unittest.mock import MagicMock, patch
	13	+
	14	+import pytest
	15	+from motor.common.resources.instance import NodeManagerInfo
	16	+from motor.controller.api_client.node_manager_api_client import NodeManagerApiClient
	17	+
	18	+# pylint: disable=redefined-outer-name
	19	+
	20	+
	21	+@pytest.fixture
	22	+def node_mgr():
	23	+ return NodeManagerInfo(pod_ip="10.0.0.1", port="8080")
	24	+
	25	+
	26	+@pytest.fixture
	27	+def mock_http_client():
	28	+ with patch("motor.controller.api_client.node_manager_api_client.SafeHTTPSClient") as mock_cls:
	29	+ mock_client = MagicMock()
	30	+ mock_cls.return_value = mock_client
	31	+ yield mock_client
	32	+
	33	+
	34	+def test_restart_engine_posts_restart(node_mgr, mock_http_client):
	35	+ assert NodeManagerApiClient.restart_engine(node_mgr, action="restart", instance_id=7) is True
	36	+ posted = mock_http_client.post.call_args
	37	+ assert posted.args[0] == "/node-manager/engine-restart"
	38	+ assert posted.kwargs["data"] == {"action": "restart", "instance_id": 7}
	39	+
	40	+
	41	+def test_restart_engine_abort_payload(node_mgr, mock_http_client):
	42	+ assert NodeManagerApiClient.restart_engine(node_mgr, action="abort") is True
	43	+ posted = mock_http_client.post.call_args
	44	+ assert posted.kwargs["data"] == {"action": "abort"}
	45	+
	46	+
	47	+def test_restart_engine_failure_returns_false(node_mgr, mock_http_client):
	48	+ mock_http_client.post.side_effect = RuntimeError("unreachable")
	49	+ assert NodeManagerApiClient.restart_engine(node_mgr, action="restart", instance_id=7) is False
	50	+
	51	+
	52	+def test_restart_engine_client_construction_failure_returns_false(node_mgr):
	53	+ """A SafeHTTPSClient construction failure must not raise UnboundLocalError in close()."""
	54	+ with patch(
	55	+ "motor.controller.api_client.node_manager_api_client.SafeHTTPSClient",
	56	+ side_effect=RuntimeError("tls config invalid"),
	57	+ ):
	58	+ assert NodeManagerApiClient.restart_engine(node_mgr, action="restart") is False

Mtests/controller/core/test_controller_config.py+31-0

	@@ -823,3 +823,34 @@ def test_tls_config_path_strings():
823	config.mgmt_tls_config.key_file = ""	823	config.mgmt_tls_config.key_file = ""
824	assert config.mgmt_tls_config.cert_file == ""	824	assert config.mgmt_tls_config.cert_file == ""
825	assert config.mgmt_tls_config.key_file == ""	825	assert config.mgmt_tls_config.key_file == ""
		826	+
		827	+
		828	+def test_config_engine_relaunch_defaults():
		829	+ """The relaunch knobs default to sane values."""
		830	+ config = ControllerConfig()
		831	+ ft = config.fault_tolerance_config
		832	+ assert ft.enable_engine_relaunch is True
		833	+ assert ft.engine_relaunch_complete_timeout_sec == 600
		834	+ assert ft.engine_relaunch_poll_interval_sec == 5.0
		835	+ assert ft.engine_relaunch_dispatch_retries == 3
		836	+ assert ft.engine_relaunch_nm_unreachable_threshold == 3
		837	+
		838	+
		839	+@pytest.mark.parametrize(
		840	+ "attr,value,expected",
		841	+ [
		842	+ ("engine_relaunch_complete_timeout_sec", 30, "engine_relaunch_complete_timeout_sec must be in range 60-3600"),
		843	+ ("engine_relaunch_poll_interval_sec", 0, "engine_relaunch_poll_interval_sec must be in range 1-60"),
		844	+ ("engine_relaunch_dispatch_retries", 11, "engine_relaunch_dispatch_retries must be in range 0-10"),
		845	+ (
		846	+ "engine_relaunch_nm_unreachable_threshold",
		847	+ 0,
		848	+ "engine_relaunch_nm_unreachable_threshold must be in range 1-10",
		849	+ ),
		850	+ ],
		851	+)
		852	+def test_config_engine_relaunch_validation(attr, value, expected):
		853	+ with pytest.raises(ValueError, match=expected):
		854	+ config = ControllerConfig()
		855	+ setattr(config.fault_tolerance_config, attr, value)
		856	+ config.validate_config()

Mtests/controller/core/test_instance_manager.py+97-0

	@@ -1284,3 +1284,100 @@ def test_restore_data_with_malformed_numeric_data():
1284		1284
1285	# Instance should not be created due to validation error	1285	# Instance should not be created due to validation error
1286	assert len(manager.instances) == 0	1286	assert len(manager.instances) == 0
		1287	+
		1288	+
		1289	+def test_stale_heartbeat_blocks_reactivation(instance_manager):
		1290	+ """All endpoints NORMAL by status, but one endpoint's heartbeat timed out:
		1291	+ the surviving NM's heartbeat must NOT flip the instance ACTIVE.
		1292	+ """
		1293	+ manager = create_instance_manager_with_config()
		1294	+ instance = create_test_instance(301, "test_stale", ["192.168.1.1", "192.168.1.2"])
		1295	+ manager.add_instance(instance)
		1296	+ instance.update_instance_status(InsStatus.INACTIVE)
		1297	+
		1298	+ for ep in instance.endpoints["192.168.1.1"].values():
		1299	+ ep.status = EndpointStatus.NORMAL
		1300	+ ep.hb_timestamp = time.time() - 60 # NM 1 is gone
		1301	+ for ep in instance.endpoints["192.168.1.2"].values():
		1302	+ ep.status = EndpointStatus.NORMAL
		1303	+ ep.hb_timestamp = time.time() # NM 2 alive
		1304	+
		1305	+ result = manager._handle_state_transition(instance)
		1306	+
		1307	+ assert result is True
		1308	+ assert instance.status == InsStatus.INACTIVE, (
		1309	+ f"Expected INACTIVE but got {instance.status} — stale heartbeats must block ACTIVE"
		1310	+ )
		1311	+
		1312	+
		1313	+def test_fresh_heartbeats_still_activate(instance_manager):
		1314	+ """With all heartbeats fresh, the NORMAL transition works as before."""
		1315	+ manager = create_instance_manager_with_config()
		1316	+ instance = create_test_instance(302, "test_fresh", ["192.168.1.1", "192.168.1.2"])
		1317	+ manager.add_instance(instance)
		1318	+ instance.update_instance_status(InsStatus.INACTIVE)
		1319	+
		1320	+ for ep in instance.endpoints["192.168.1.1"].values():
		1321	+ ep.status = EndpointStatus.NORMAL
		1322	+ ep.hb_timestamp = time.time()
		1323	+ for ep in instance.endpoints["192.168.1.2"].values():
		1324	+ ep.status = EndpointStatus.NORMAL
		1325	+ ep.hb_timestamp = time.time()
		1326	+
		1327	+ result = manager._handle_state_transition(instance)
		1328	+
		1329	+ assert result is True
		1330	+ assert instance.status == InsStatus.ACTIVE
		1331	+
		1332	+
		1333	+def test_check_node_managers_status_probes_all_before_returning(instance_manager):
		1334	+ """An abnormal NM earlier in the list must not drop the later reachable survivors."""
		1335	+ manager = create_instance_manager_with_config()
		1336	+ instance = create_test_instance(305, "test_probe_all", ["192.168.1.1", "192.168.1.2"])
		1337	+ manager.add_instance(instance)
		1338	+
		1339	+ nm_abnormal = MagicMock(pod_ip="192.168.1.1", port="8080")
		1340	+ nm_normal = MagicMock(pod_ip="192.168.1.2", port="8080")
		1341	+ with (
		1342	+ patch.object(Instance, "get_node_managers", return_value=[nm_abnormal, nm_normal]),
		1343	+ patch("motor.controller.core.instance_manager.NodeManagerApiClient") as mock_client_cls,
		1344	+ ):
		1345	+ mock_client_cls.query_status.side_effect = [{"status": False}, {"status": True}]
		1346	+
		1347	+ has_abnormal, reachable = manager._check_node_managers_status(instance)
		1348	+
		1349	+ assert has_abnormal is True
		1350	+ # the survivor behind the abnormal NM is still discovered
		1351	+ assert reachable == [nm_normal]
		1352	+
		1353	+
		1354	+def test_partial_loss_dispatches_stop_with_dedup(instance_manager):
		1355	+ """Partial loss dispatches stop to the reachable NM once per episode; the
		1356	+ dedup marker clears when the instance recovers to ACTIVE.
		1357	+ """
		1358	+ manager = create_instance_manager_with_config()
		1359	+ instance = create_test_instance(303, "test_loss", ["192.168.1.1", "192.168.1.2"])
		1360	+ manager.add_instance(instance)
		1361	+ instance.update_instance_status(InsStatus.ACTIVE)
		1362	+
		1363	+ reachable = [MagicMock(pod_ip="192.168.1.2", port="8080")]
		1364	+ with (
		1365	+ patch.object(manager, "_check_node_managers_status", return_value=(True, reachable)),
		1366	+ patch("motor.controller.core.instance_manager.NodeManagerApiClient") as mock_client_cls,
		1367	+ ):
		1368	+ mock_client_cls.stop.return_value = True
		1369	+
		1370	+ manager._handle_inactive(InsStatus.ACTIVE, InsConditionEvent.INSTANCE_HEARTBEAT_TIMEOUT, instance)
		1371	+ assert mock_client_cls.stop.call_count == 1
		1372	+ mock_client_cls.stop.assert_called_once_with(reachable[0])
		1373	+
		1374	+ # second episode round in the same episode: deduped
		1375	+ manager._handle_inactive(InsStatus.ACTIVE, InsConditionEvent.INSTANCE_HEARTBEAT_TIMEOUT, instance)
		1376	+ assert mock_client_cls.stop.call_count == 1
		1377	+
		1378	+ # instance recovers (INACTIVE -> ACTIVE via _handle_active)
		1379	+ manager._handle_active(InsStatus.INACTIVE, InsConditionEvent.INSTANCE_NORMAL, instance)
		1380	+
		1381	+ # a new loss episode dispatches again
		1382	+ manager._handle_inactive(InsStatus.ACTIVE, InsConditionEvent.INSTANCE_HEARTBEAT_TIMEOUT, instance)
		1383	+ assert mock_client_cls.stop.call_count == 2

Atests/controller/fault_tolerance/strategy/test_engine_relaunch.py+203-0

@@ -0,0 +1,203 @@
	1	+# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
	2	+# MindIE is licensed under Mulan PSL v2.
	3	+# You can use this software according to the terms and conditions of the Mulan PSL v2.
	4	+# You may obtain a copy of Mulan PSL v2 at:
	5	+# http://license.coscl.org.cn/MulanPSL2
	6	+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
	7	+# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
	8	+# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
	9	+# See the Mulan PSL v2 for more details.
	10	+"""Tests for the EngineRelaunchStrategy two-phase fallback recovery."""
	11	+
	12	+from contextlib import contextmanager
	13	+from unittest.mock import Mock, patch
	14	+
	15	+import pytest
	16	+
	17	+from motor.common.resources.instance import InsStatus, NodeManagerInfo
	18	+from motor.controller.fault_tolerance.strategy.engine_relaunch import (
	19	+ EngineRelaunchStrategy,
	20	+ RelaunchState,
	21	+)
	22	+
	23	+# pylint: disable=redefined-outer-name
	24	+
	25	+
	26	+def _make_instance(node_managers, instance_id=1, status=InsStatus.INACTIVE):
	27	+ instance = Mock()
	28	+ instance.id = instance_id
	29	+ instance.status = status
	30	+ instance.get_node_managers.return_value = node_managers
	31	+ return instance
	32	+
	33	+
	34	+@pytest.fixture
	35	+def nm_a():
	36	+ return NodeManagerInfo(pod_ip="10.0.0.1", port="8080")
	37	+
	38	+
	39	+@pytest.fixture
	40	+def nm_b():
	41	+ return NodeManagerInfo(pod_ip="10.0.0.2", port="8080")
	42	+
	43	+
	44	+@contextmanager
	45	+def _apply_patch():
	46	+ """Mock InstanceManager, the client and the FaultManager config singleton."""
	47	+ with (
	48	+ patch("motor.controller.core.instance_manager.InstanceManager") as mock_im_cls,
	49	+ patch("motor.controller.fault_tolerance.strategy.engine_relaunch.NodeManagerApiClient") as mock_client,
	50	+ patch("motor.controller.fault_tolerance.fault_manager.FaultManager") as mock_fm_cls,
	51	+ ):
	52	+ ft_config = Mock()
	53	+ ft_config.engine_relaunch_complete_timeout_sec = 600
	54	+ ft_config.engine_relaunch_poll_interval_sec = 0.01
	55	+ ft_config.engine_relaunch_dispatch_retries = 3
	56	+ ft_config.engine_relaunch_nm_unreachable_threshold = 3
	57	+ mock_fm_cls.return_value.config.fault_tolerance_config = ft_config
	58	+ mock_client.restart_engine.return_value = True
	59	+ yield mock_im_cls, mock_client
	60	+
	61	+
	62	+def _new_strategy() -> EngineRelaunchStrategy:
	63	+ strategy = EngineRelaunchStrategy()
	64	+ strategy.engine_relaunch_poll_interval_sec = 0.01
	65	+ return strategy
	66	+
	67	+
	68	+def test_execute_dispatch_and_poll_until_all_normal(nm_a, nm_b):
	69	+ """Every NodeManager gets the restart command; polling continues until all recover."""
	70	+ instance = _make_instance([nm_a, nm_b])
	71	+ with _apply_patch() as (mock_im_cls, mock_client):
	72	+ mock_im_cls.return_value.get_instance.return_value = instance
	73	+ mock_client.query_status.side_effect = [
	74	+ {"status": False}, # probe nm_a
	75	+ {"status": False}, # probe nm_b
	76	+ {"status": False}, # poll round 1: nm_a loading
	77	+ {"status": False}, # poll round 1: nm_b loading
	78	+ {"status": True}, # poll round 2: nm_a ready
	79	+ {"status": True}, # poll round 2: nm_b ready
	80	+ ]
	81	+
	82	+ strategy = _new_strategy()
	83	+ strategy.execute(1)
	84	+
	85	+ assert strategy.is_finished()
	86	+ assert strategy.context.current_state == RelaunchState.SUCCESS
	87	+ restart_calls = [
	88	+ c for c in mock_client.restart_engine.call_args_list if c.kwargs.get("action", "restart") == "restart"
	89	+ ]
	90	+ assert len(restart_calls) == 2
	91	+ for call in restart_calls:
	92	+ assert call.kwargs["instance_id"] == 1
	93	+ assert mock_client.query_status.call_count >= 6
	94	+
	95	+
	96	+@pytest.mark.parametrize(
	97	+ "status_script, expect_dispatched, expect_aborted",
	98	+ [
	99	+ # probe fails -> nothing dispatched -> Phase 2 aborts it
	100	+ ([RuntimeError("unreachable")], False, True),
	101	+ # poll fails after dispatch -> already relaunching -> freeze kept, no abort
	102	+ ([{"status": False}, RuntimeError("gone")], True, False),
	103	+ ],
	104	+ ids=["probe_fails", "poll_fails"],
	105	+)
	106	+def test_execute_unreachable_nm_escalates_to_container_restart(nm_a, status_script, expect_dispatched, expect_aborted):
	107	+ """An unreachable NodeManager escalates to Phase 2.
	108	+
	109	+ Phase 2 only aborts the NodeManagers that were NOT dispatched: one that
	110	+ already accepted the restart is relaunching in place and keeps its
	111	+ suicide freeze (the deadline provides the fallback).
	112	+ """
	113	+ instance = _make_instance([nm_a])
	114	+ with _apply_patch() as (mock_im_cls, mock_client):
	115	+ mock_im_cls.return_value.get_instance.return_value = instance
	116	+ mock_client.query_status.side_effect = status_script
	117	+
	118	+ strategy = _new_strategy()
	119	+ strategy.engine_relaunch_nm_unreachable_threshold = 3
	120	+ strategy.execute(1)
	121	+
	122	+ assert strategy.context.current_state == RelaunchState.FAILED
	123	+ assert strategy.is_failed()
	124	+ restart_calls = [
	125	+ c for c in mock_client.restart_engine.call_args_list if c.kwargs.get("action", "restart") == "restart"
	126	+ ]
	127	+ assert bool(restart_calls) is expect_dispatched
	128	+ abort_calls = [c for c in mock_client.restart_engine.call_args_list if c.kwargs.get("action") == "abort"]
	129	+ assert bool(abort_calls) is expect_aborted
	130	+
	131	+
	132	+def test_execute_timeout_escalates_but_keeps_dispatched_freeze(nm_a):
	133	+ """Completion timeout escalates to Phase 2, but the dispatched NM keeps its freeze."""
	134	+ instance = _make_instance([nm_a])
	135	+ with _apply_patch() as (mock_im_cls, mock_client):
	136	+ mock_im_cls.return_value.get_instance.return_value = instance
	137	+ mock_client.query_status.return_value = {"status": False}
	138	+
	139	+ strategy = _new_strategy()
	140	+ strategy.engine_relaunch_complete_timeout_sec = 0.02
	141	+ strategy.execute(1)
	142	+
	143	+ assert strategy.context.current_state == RelaunchState.FAILED
	144	+ assert strategy.is_failed()
	145	+ abort_calls = [c for c in mock_client.restart_engine.call_args_list if c.kwargs.get("action") == "abort"]
	146	+ assert len(abort_calls) == 0
	147	+ assert strategy.context.dispatch_results.get(nm_a.pod_ip) is True
	148	+
	149	+
	150	+def test_execute_partial_dispatch_aborts_only_failed_nm(nm_a, nm_b):
	151	+ """Phase 2 aborts only the NM whose restart dispatch failed."""
	152	+ instance = _make_instance([nm_a, nm_b])
	153	+ with _apply_patch() as (mock_im_cls, mock_client):
	154	+ mock_im_cls.return_value.get_instance.return_value = instance
	155	+ mock_client.query_status.return_value = {"status": False}
	156	+ mock_client.restart_engine.side_effect = [True, False] # nm_a ok, nm_b fails
	157	+
	158	+ strategy = _new_strategy()
	159	+ strategy.engine_relaunch_complete_timeout_sec = 0.02
	160	+ strategy.engine_relaunch_dispatch_retries = 1
	161	+ strategy.execute(1)
	162	+
	163	+ assert strategy.context.current_state == RelaunchState.FAILED
	164	+ assert strategy.context.dispatch_results == {nm_a.pod_ip: True, nm_b.pod_ip: False}
	165	+ abort_calls = [c for c in mock_client.restart_engine.call_args_list if c.kwargs.get("action") == "abort"]
	166	+ assert [c.args[0].pod_ip for c in abort_calls] == [nm_b.pod_ip]
	167	+
	168	+
	169	+def test_execute_instance_deleted_finishes_without_fallback(nm_a):
	170	+ """Instance disappearing mid-relaunch ends the strategy (delete flow takes over)."""
	171	+ instance = _make_instance([nm_a])
	172	+
	173	+ def _instances():
	174	+ instance.status = InsStatus.DELETED
	175	+ return instance
	176	+
	177	+ with _apply_patch() as (mock_im_cls, mock_client):
	178	+ mock_im_cls.return_value.get_instance.side_effect = [instance, _instances()]
	179	+ mock_client.query_status.return_value = {"status": False}
	180	+
	181	+ strategy = _new_strategy()
	182	+ strategy.execute(1)
	183	+
	184	+ assert strategy.context.current_state == RelaunchState.SUCCESS
	185	+ assert not strategy.is_failed()
	186	+
	187	+
	188	+def test_stop_sets_event_and_aborts(nm_a):
	189	+ """stop() interrupts the wait and unfreezes the NodeManagers."""
	190	+ instance = _make_instance([nm_a])
	191	+ with _apply_patch() as (mock_im_cls, mock_client):
	192	+ mock_im_cls.return_value.get_instance.return_value = instance
	193	+ mock_client.query_status.return_value = {"status": False}
	194	+
	195	+ strategy = _new_strategy()
	196	+ strategy.context = Mock(instance_id=1, node_managers=[nm_a])
	197	+ strategy.stop()
	198	+
	199	+ import time
	200	+
	201	+ time.sleep(0.2) # fire-and-forget abort thread
	202	+ abort_calls = [c for c in mock_client.restart_engine.call_args_list if c.kwargs.get("action") == "abort"]
	203	+ assert len(abort_calls) == 1

Mtests/controller/fault_tolerance/strategy/test_strategy.py+31-1

	@@ -29,7 +29,7 @@ Test cases are organized according to the following logical blocks:
29	import pytest	29	import pytest
30	from unittest.mock import Mock, patch	30	from unittest.mock import Mock, patch
31		31
32	-from motor.controller.fault_tolerance.fault_types import FaultLevel	32	+from motor.controller.fault_tolerance.fault_types import FaultLevel, SpecialFaultCode
33	from motor.controller.fault_tolerance.strategy.strategy import (	33	from motor.controller.fault_tolerance.strategy.strategy import (
34	StrategyBase,	34	StrategyBase,
35	healthy_strategy,	35	healthy_strategy,
	@@ -329,3 +329,33 @@ def test_level6_strategy_respects_config_switch(mock_instance_manager, decode_in
329		329
330	result = level6_strategy(0x0000, 1, mock_config_scale_p2d_disabled)	330	result = level6_strategy(0x0000, 1, mock_config_scale_p2d_disabled)
331	assert result is None	331	assert result is None
		332	+
		333	+
		334	+# -- engine relaunch (fallback) hooking ----------------------------------------
		335	+
		336	+
		337	+def test_level2_engine_dead_returns_engine_relaunch(mock_config):
		338	+ """ENGINE_DEAD (L2 software fault) escalates to the relaunch fallback."""
		339	+ from motor.controller.fault_tolerance.strategy.engine_relaunch import EngineRelaunchStrategy
		340	+
		341	+ assert level2_strategy(int(SpecialFaultCode.ENGINE_DEAD), 1, mock_config) is EngineRelaunchStrategy
		342	+
		343	+
		344	+def test_level2_engine_dead_respects_switch(mock_config):
		345	+ """enable_engine_relaunch=False restores the legacy no-op behavior."""
		346	+ mock_config.fault_tolerance_config.enable_engine_relaunch = False
		347	+ assert level2_strategy(int(SpecialFaultCode.ENGINE_DEAD), 1, mock_config) is None
		348	+
		349	+
		350	+def test_level2_engine_unhealthy_stays_none(mock_config):
		351	+ """ENGINE_UNHEALTHY keeps the current no-op behavior (fast-recovery domain)."""
		352	+ assert level2_strategy(int(SpecialFaultCode.ENGINE_UNHEALTHY), 1, mock_config) is None
		353	+
		354	+
		355	+def test_strategy_base_failed_semantics():
		356	+ """mark_failed/is_failed round-trip; finished stays independent."""
		357	+ strategy = TokenReinferenceStrategy()
		358	+ assert not strategy.is_failed()
		359	+ strategy.mark_failed()
		360	+ assert strategy.is_failed()
		361	+ assert not strategy.is_finished()

Mtests/controller/fault_tolerance/test_fault_manager.py+67-3

     )
     # ManuallySeparateNPU is NOT PreSeparateNPU → filter returns True
++    assert _affects_instance(FAULT_MANUALLY_SEPARATE_L6.model_copy(), node_with_instances) is True, (
         "ManuallySeparateNPU L6 should always be included"
     )
     # PreSeparateNPU L6: included when instance still on node
++    assert _affects_instance(FAULT_PRE_SEPARATE_L6.model_copy(), node_with_instances) is True, (
         "PreSeparateNPU L6 should be included when instances remain on node"
     )
     # PreSeparateNPU L6: excluded when no instances on node (already moved)
++    assert _affects_instance(FAULT_PRE_SEPARATE_L6.model_copy(), node_no_instances) is False, (
         "PreSeparateNPU L6 should be excluded when instance has left the node"
     )
     with patch(_CORE_IM) as mock_im_class:
         mock_im_class.return_value = mock_im
         assert fault_manager._node_has_active_instances(node) is True
++
++
++def test_process_instance_strategy_escalates_after_failed_strategy(fault_manager_with_instances):
++    """A failed strategy escalates to the EngineRelaunch fallback next round."""
++    manager = fault_manager_with_instances
++    manager.instances[1].fault_level = FaultLevel.L2
++    manager.instances[1].fault_code = int(SpecialFaultCode.ENGINE_DEAD)
++    manager.instances[1].prev_strategy_failed = True
++    manager.instances[1].strategy = None
++
++    # A running (not yet finished) failed-strategy marker must trigger the fallback selection.
++    # executor.submit is mocked so the strategy stays "running" (is_finished=False) — without
++    # the mock the strategy would execute and finish instantly (mocked InstanceManager) and the
++    # completion hook would clear it, making this assertion race the executor thread.
++    with (
++        patch("motor.controller.core.instance_manager.InstanceManager") as mock_im_class,
++        patch.object(manager.executor, "submit") as mock_submit,
++    ):
++        mock_im = MagicMock()
++        mock_im_class.return_value = mock_im
++
++        manager._process_instance_strategy(1)
++
++        from motor.controller.fault_tolerance.strategy.engine_relaunch import EngineRelaunchStrategy
++
++        mock_submit.assert_called_once()
++        assert manager.instances[1].strategy is not None
++        assert isinstance(manager.instances[1].strategy, EngineRelaunchStrategy)
++
++
++def test_strategy_completion_records_failure_flag(fault_manager_with_instances):
++    """A finished failed strategy marks prev_strategy_failed on the instance."""
++    manager = fault_manager_with_instances
++
++    failed_strategy = MagicMock()
++    failed_strategy.is_finished.return_value = True
++    failed_strategy.is_failed.return_value = True
++    manager.instances[1].strategy = failed_strategy
++    manager.instances[1].strategy_fault_level = FaultLevel.L2
++
++    with patch("motor.controller.fault_tolerance.fault_manager.InstanceManager") as mock_im_class:
++        mock_im_class.return_value = MagicMock()
++        manager._process_instance_strategy(1)
++
++    assert manager.instances[1].prev_strategy_failed is True
++    assert manager.instances[1].strategy is None
++
++
++def test_strategy_completion_clears_failure_flag(fault_manager_with_instances):
++    """A finished successful strategy clears prev_strategy_failed."""
++    manager = fault_manager_with_instances
++
++    ok_strategy = MagicMock()
++    ok_strategy.is_finished.return_value = True
++    ok_strategy.is_failed.return_value = False
++    manager.instances[1].strategy = ok_strategy
++    manager.instances[1].strategy_fault_level = FaultLevel.L2
++    manager.instances[1].prev_strategy_failed = True
++
++    with patch("motor.controller.fault_tolerance.fault_manager.InstanceManager") as mock_im_class:
++        mock_im_class.return_value = MagicMock()
++        manager._process_instance_strategy(1)
++
++    assert manager.instances[1].prev_strategy_failed is False

Atests/node_manager/api_server/test_api_server.py+135-0

@@ -0,0 +1,135 @@
	1	+# Copyright (c) Huawei Technologies Co., Ltd. 2025-2026. All rights reserved.
	2	+# MindIE is licensed under Mulan PSL v2.
	3	+# You can use this software according to the terms and conditions of the Mulan PSL v2.
	4	+# You may obtain a copy of Mulan PSL v2 at:
	5	+# http://license.coscl.org.cn/MulanPSL2
	6	+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND,
	7	+# EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT,
	8	+# MERCHANTABILITY OR FIT FOR A PARTICULAR PURPOSE.
	9	+# See the Mulan PSL v2 for more details.
	10	+"""Tests for the NodeManager API server routes (engine-restart, stop, ...)."""
	11	+
	12	+import os
	13	+import sys
	14	+
	15	+import pytest
	16	+from fastapi.testclient import TestClient
	17	+from unittest.mock import patch
	18	+
	19	+os.environ["USER_CONFIG_PATH"] = "tests/jsons/useruser_config.json"
	20	+os.environ["ROLE"] = "both"
	21	+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
	22	+
	23	+from motor.node_manager.api_server.node_manager_api import app
	24	+
	25	+# pylint: disable=redefined-outer-name
	26	+
	27	+
	28	+@pytest.fixture
	29	+def client():
	30	+ return TestClient(app)
	31	+
	32	+
	33	+@pytest.fixture
	34	+def route_mocks():
	35	+ """Patch the singleton components the route talks to."""
	36	+ with (
	37	+ patch("motor.node_manager.api_server.node_manager_api.HeartbeatManager") as mock_hb_cls,
	38	+ patch("motor.node_manager.api_server.node_manager_api.Daemon") as mock_daemon_cls,
	39	+ ):
	40	+ yield mock_hb_cls, mock_daemon_cls
	41	+
	42	+
	43	+@pytest.mark.parametrize(
	44	+ "payload",
	45	+ ["nope", {"action": "explode"}, {"action": "shutdown"}],
	46	+ ids=["invalid-json", "unknown-action", "removed-shutdown-action"],
	47	+)
	48	+def test_engine_restart_rejects_invalid_requests(client, route_mocks, payload):
	49	+ resp = client.post("/node-manager/engine-restart", json=payload)
	50	+ assert resp.status_code == 400
	51	+
	52	+
	53	+def test_engine_restart_abort_unfreezes(client, route_mocks):
	54	+ _, mock_daemon_cls = route_mocks
	55	+ resp = client.post("/node-manager/engine-restart", json={"action": "abort"})
	56	+ assert resp.status_code == 200
	57	+ mock_daemon_cls.return_value.unfreeze_suicide.assert_called_once()
	58	+
	59	+
	60	+@pytest.mark.parametrize(
	61	+ "instance_id,started_after_restore,expected",
	62	+ [(7, True, 7), (None, False, None)],
	63	+ ids=["with-instance-id", "no-instance-id"],
	64	+)
	65	+def test_engine_restart_dispatch(client, route_mocks, instance_id, started_after_restore, expected):
	66	+ """restart delegates the whole relaunch to the Daemon (params resolved there)."""
	67	+ mock_hb_cls, mock_daemon_cls = route_mocks
	68	+ mock_hb_cls.return_value.is_started_after_restore.return_value = started_after_restore
	69	+
	70	+ resp = client.post("/node-manager/engine-restart", json={"action": "restart", "instance_id": instance_id})
	71	+
	72	+ assert resp.status_code == 200
	73	+ mock_daemon_cls.return_value.restart_engine.assert_called_once_with(expected)
	74	+
	75	+
	76	+def test_engine_restart_no_start_recorded(client, route_mocks):
	77	+ """No launch params recorded: the Daemon raises and the route maps it to 400."""
	78	+ from motor.node_manager.core.daemon import EngineRestartParamError
	79	+
	80	+ mock_hb_cls, mock_daemon_cls = route_mocks
	81	+ mock_hb_cls.return_value.is_started_after_restore.return_value = True
	82	+ mock_daemon_cls.return_value.restart_engine.side_effect = EngineRestartParamError()
	83	+
	84	+ resp = client.post("/node-manager/engine-restart", json={"action": "restart"})
	85	+ assert resp.status_code == 400
	86	+
	87	+
	88	+def test_engine_restart_rejects_snapshot_restore_in_progress(client, route_mocks):
	89	+ """409 only during an actual snapshot restore — normal deploys must relaunch."""
	90	+ mock_hb_cls, mock_daemon_cls = route_mocks
	91	+ mock_hb_cls.return_value.is_started_after_restore.return_value = False
	92	+
	93	+ with patch("motor.node_manager.api_server.node_manager_api.is_restored_from_host_side_snapshot", return_value=True):
	94	+ resp = client.post("/node-manager/engine-restart", json={"action": "restart"})
	95	+
	96	+ assert resp.status_code == 409
	97	+ mock_daemon_cls.return_value.restart_engine.assert_not_called()
	98	+
	99	+
	100	+def test_engine_restart_concurrent_rejected(client, route_mocks):
	101	+ """A second restart while one is in progress is rejected with 409."""
	102	+ from motor.node_manager.core.daemon import EngineRestartInProgressError
	103	+
	104	+ mock_hb_cls, mock_daemon_cls = route_mocks
	105	+ mock_hb_cls.return_value.is_started_after_restore.return_value = True
	106	+ mock_daemon_cls.return_value.restart_engine.side_effect = EngineRestartInProgressError()
	107	+
	108	+ resp = client.post("/node-manager/engine-restart", json={"action": "restart"})
	109	+ assert resp.status_code == 409
	110	+
	111	+
	112	+def test_engine_restart_pull_failure_returns_500(client, route_mocks):
	113	+ """A failed engine pull surfaces as 500 (suicide unfreeze is the Daemon's job)."""
	114	+ mock_hb_cls, mock_daemon_cls = route_mocks
	115	+ mock_hb_cls.return_value.is_started_after_restore.return_value = True
	116	+ mock_daemon_cls.return_value.restart_engine.side_effect = RuntimeError("pull failed")
	117	+
	118	+ resp = client.post("/node-manager/engine-restart", json={"action": "restart"})
	119	+
	120	+ assert resp.status_code == 500
	121	+
	122	+
	123	+def test_node_manager_stop_sigterms_self(client, route_mocks):
	124	+ """stop kills the engines and schedules a delayed self-termination (pod restart)."""
	125	+ _, mock_daemon_cls = route_mocks
	126	+ with (
	127	+ patch("motor.node_manager.api_server.node_manager_api.threading.Timer") as mock_timer_cls,
	128	+ patch("motor.node_manager.api_server.node_manager_api._self_terminate") as mock_terminate,
	129	+ ):
	130	+ resp = client.post("/node-manager/stop", json={})
	131	+
	132	+ assert resp.status_code == 200
	133	+ mock_daemon_cls.return_value.stop.assert_called_once()
	134	+ mock_timer_cls.assert_called_once_with(0.5, mock_terminate)
	135	+ mock_timer_cls.return_value.start.assert_called_once()

Mtests/node_manager/core/services/native_engine/test_service.py+7-23

	@@ -11,10 +11,7 @@
11	from types import SimpleNamespace	11	from types import SimpleNamespace
12	from unittest.mock import patch	12	from unittest.mock import patch
13		13
14	-from motor.common.resources.endpoint import Endpoint
15	-from motor.common.resources.instance import PDRole
16	from motor.node_manager.core.services.native_engine.service import NativeEngineService	14	from motor.node_manager.core.services.native_engine.service import NativeEngineService
17	-from motor.node_manager.core.services.native_engine.models import CommandSpec, LaunchSpec, ProbeSpec
18		15
19		16
20	def _native_engine_service() -> NativeEngineService:	17	def _native_engine_service() -> NativeEngineService:
	@@ -34,29 +31,16 @@ def _native_engine_service() -> NativeEngineService:
34	return service	31	return service
35		32
36		33
37	-def test_successful_pull_clears_recovery_latch():	34	+def test_health_check_returns_dead_pids():
		35	+ """health_check only surfaces dead PIDs; recovery routing is the Daemon's."""
38	service = _native_engine_service()	36	service = _native_engine_service()
39	- service._recovery_requested = True	37	+ service.supervisor.dead_pids.return_value = [(101, 0)]
40	- service.backend.prepare.return_value = LaunchSpec(
41	- command=CommandSpec(argv=("python", "-m", "sglang.launch_server"), env={}),
42	- probe=ProbeSpec(path="/health", timeout_seconds=1, startup_timeout_seconds=10),
43	- )
44	- service.supervisor.start.return_value = True
45	- endpoint = Endpoint(id=0, ip="127.0.0.1", business_port="8000", mgmt_port="8001")
46		38
47	- service.pull(PDRole.ROLE_D, [endpoint], instance_id=1, master_dp_ip="127.0.0.1")	39	+ assert service.health_check() == [(101, 0)]
48	-
49	- assert service._recovery_requested is False
50	- service.supervisor.start.assert_called_once()
51		40
52		41
53	-@patch("motor.node_manager.core.services.native_engine.service.os.kill")	42	+def test_health_check_empty_when_no_deaths():
54	-def test_health_check_requests_recovery_only_once(mock_kill):
55	service = _native_engine_service()	43	service = _native_engine_service()
56	- service.restart_on_failure = True	44	+ service.supervisor.dead_pids.return_value = []
57	- service.supervisor.dead_pids.side_effect = [[101], [102]]
58		45
59	- service.health_check()	46	+ assert service.health_check() == []
60	- service.health_check()
61	-
62	- mock_kill.assert_called_once()

Mtests/node_manager/core/services/native_engine/test_supervisor.py+1-1

	@@ -245,7 +245,7 @@ def test_dead_process_is_stopped_without_http_probe(mock_popen):
245		245
246	assert supervisor.state(3, "10.0.0.1", 8000) == RuntimeState.STOPPED	246	assert supervisor.state(3, "10.0.0.1", 8000) == RuntimeState.STOPPED
247	with patch.object(supervisor, "_kill_group") as kill_group:	247	with patch.object(supervisor, "_kill_group") as kill_group:
248	- assert supervisor.dead_pids() == [12345]	248	+ assert supervisor.dead_pids() == [(12345, 3)]
249	kill_group.assert_called_once()	249	kill_group.assert_called_once()
250	assert supervisor.dead_pids() == []	250	assert supervisor.dead_pids() == []
251	assert supervisor.pid_list() == []	251	assert supervisor.pid_list() == []

Mtests/node_manager/core/test_daemon.py+269-30

	@@ -10,10 +10,11 @@
10		10
11	import os	11	import os
12	import json	12	import json
13	-import signal
14	import pytest	13	import pytest
15	from unittest.mock import patch, MagicMock, mock_open	14	from unittest.mock import patch, MagicMock, mock_open
16		15
		16	+os.environ["USER_CONFIG_PATH"] = os.path.join(os.path.dirname(__file__), "..", "jsons", "user_config.json")
		17	+
17	from motor.node_manager.core.daemon import Daemon	18	from motor.node_manager.core.daemon import Daemon
18	from motor.node_manager.core.services.registry import SERVICE_ENGINE	19	from motor.node_manager.core.services.registry import SERVICE_ENGINE
19	from motor.config.node_manager import NodeManagerConfig	20	from motor.config.node_manager import NodeManagerConfig
	@@ -175,35 +176,6 @@ class TestDaemon:
175		176
176	stop.assert_called_once_with(endpoints[1].id)	177	stop.assert_called_once_with(endpoints[1].id)
177		178
178	- @pytest.mark.parametrize(
179	- "invalid_endpoint,error_msg",
180	- [
181	- (Endpoint(id=0, ip="invalid_ip", business_port="8000", mgmt_port="9090"), "Failed to pull engine"),
182	- (Endpoint(id=0, ip="192.168.1.1", business_port="999999", mgmt_port="9090"), "Failed to pull engine"),
183	- ],
184	- )
185	- def test_pull_engine_invalid_params(self, daemon, invalid_endpoint, error_msg):
186	- with pytest.raises(RuntimeError, match=error_msg):
187	- daemon.pull_engine(PDRole.ROLE_U, [invalid_endpoint], instance_id=1, master_dp_ip="192.168.1.100")
188	-
189	- def test_exit_daemon_delegates_to_process_supervisor(self, daemon):
190	- supervisor = daemon._services[SERVICE_ENGINE].supervisor
191	- with patch.object(supervisor, "stop_all", return_value=[1001, 1002]) as stop_all:
192	- daemon.stop()
193	- stop_all.assert_called_once_with()
194	-
195	- def test_engine_exit_requests_pod_recovery_only_once(self, daemon):
196	- engine = daemon._services[SERVICE_ENGINE]
197	- engine.restart_on_failure = True
198	- with (
199	- patch.object(engine.supervisor, "dead_pids", return_value=[12345]),
200	- patch("motor.node_manager.core.services.native_engine.service.os.kill") as kill,
201	- ):
202	- engine.health_check()
203	- engine.health_check()
204	-
205	- kill.assert_called_once_with(os.getpid(), signal.SIGTERM)
206	-
207	@patch('subprocess.Popen')	179	@patch('subprocess.Popen')
208	def test_native_metrics_target_uses_business_port(self, mock_popen, daemon):	180	def test_native_metrics_target_uses_business_port(self, mock_popen, daemon):
209	process = MagicMock(pid=12345)	181	process = MagicMock(pid=12345)
	@@ -418,3 +390,270 @@ class TestDaemon:
418	daemon.pull_engine(PDRole.ROLE_P, [endpoint], instance_id=1, master_dp_ip="192.168.1.100", node_rank=2)	390	daemon.pull_engine(PDRole.ROLE_P, [endpoint], instance_id=1, master_dp_ip="192.168.1.100", node_rank=2)
419		391
420	assert _last_launch_context(daemon).node_rank == 2	392	assert _last_launch_context(daemon).node_rank == 2
		393	+
		394	+
		395	+def _patch_restart_params(restart_params):
		396	+ """Patch the RegisterManager the Daemon resolves relaunch params from."""
		397	+ return patch(
		398	+ "motor.node_manager.core.register_manager.RegisterManager",
		399	+ return_value=MagicMock(get_restart_params=MagicMock(return_value=restart_params)),
		400	+ )
		401	+
		402	+
		403	+def _restart_params():
		404	+ return {
		405	+ "role": "prefill",
		406	+ "endpoints": [
		407	+ Endpoint(id=i, ip=f"192.168.1.{100 + i}", business_port=str(8000 + i * 2), mgmt_port=str(9000 + i * 2))
		408	+ for i in range(2)
		409	+ ],
		410	+ "instance_id": 1,
		411	+ "master_dp_ip": "192.168.1.100",
		412	+ "d2d_peer_ips": None,
		413	+ "node_rank": 0,
		414	+ }
		415	+
		416	+
		417	+@patch("subprocess.Popen")
		418	+def test_restart_engine_relaunches_engines_in_place(mock_popen, daemon, endpoints, capsys):
		419	+ """restart_engine only touches the engine service: KV store and monitor
		420	+ thread stay alive, the FaultReporter is paused/resumed for the window.
		421	+ """
		422	+ mock_process = MagicMock(pid=12345)
		423	+ mock_process.poll.return_value = None
		424	+ mock_popen.return_value = mock_process
		425	+
		426	+ kv_service = MagicMock()
		427	+ daemon._services["kv_store"] = kv_service
		428	+ daemon.pull_engine(PDRole.ROLE_P, endpoints, 1, "192.168.1.100")
		429	+ assert 12345 in daemon._services["engine"].pid_list()
		430	+
		431	+ fault_reporter = MagicMock()
		432	+ daemon._fault_reporter = fault_reporter
		433	+ with _patch_restart_params(_restart_params()):
		434	+ daemon.restart_engine(1)
		435	+
		436	+ kv_service.stop.assert_not_called()
		437	+ assert daemon._monitor_thread is not None and daemon._monitor_thread.is_alive()
		438	+ assert len(daemon._services["engine"].pid_list()) == len(_restart_params()["endpoints"])
		439	+ fault_reporter.pause.assert_called_once()
		440	+ fault_reporter.resume.assert_called_once()
		441	+ assert not daemon.is_engine_restart_in_progress()
		442	+ # The relaunch separator (with per-container count) hits the container
		443	+ # stdout stream so log review can delimit relaunch #N.
		444	+ engine_service = daemon._services["engine"]
		445	+ assert engine_service._restart_count == 1
		446	+ assert "[ENGINE RELAUNCH #1] instance_id=1" in capsys.readouterr().out
		447	+
		448	+
		449	+@pytest.mark.parametrize(
		450	+ "prepare,error",
		451	+ [
		452	+ (lambda d: setattr(d, "_engine_restart_in_progress", True), "EngineRestartInProgressError"),
		453	+ (lambda d: None, "EngineRestartParamError"),
		454	+ ],
		455	+ ids=["already_in_progress", "no_start_recorded"],
		456	+)
		457	+def test_restart_engine_rejects_invalid_requests(daemon, prepare, error):
		458	+ """Overlapping relaunches and missing launch params are rejected, not raced."""
		459	+ from motor.node_manager.core.daemon import EngineRestartInProgressError, EngineRestartParamError
		460	+
		461	+ expected = EngineRestartInProgressError if error == "EngineRestartInProgressError" else EngineRestartParamError
		462	+ prepare(daemon)
		463	+ with _patch_restart_params(None if error == "EngineRestartParamError" else _restart_params()):
		464	+ with pytest.raises(expected):
		465	+ daemon.restart_engine(1)
		466	+
		467	+
		468	+def test_suicide_after_threshold_abnormal_observations(daemon):
		469	+ """Continuous abnormal endpoint observations reach the threshold."""
		470	+ with (
		471	+ patch("motor.node_manager.core.daemon.HeartbeatManager") as mock_hb_cls,
		472	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		473	+ ):
		474	+ mock_hb = mock_hb_cls.return_value
		475	+ mock_hb.endpoints_generation.return_value = 0
		476	+ mock_hb.is_within_grace_period.return_value = False
		477	+ mock_hb.has_abnormal_endpoints.return_value = True
		478	+
		479	+ daemon._last_endpoints_generation = 0
		480	+ for _ in range(daemon._suicide_threshold):
		481	+ daemon._check_suicide_condition()
		482	+
		483	+ assert daemon.should_suicide() is True
		484	+
		485	+
		486	+def test_suicide_counter_reset_conditions(daemon):
		487	+ """Healthy endpoints, the grace period or a generation change reset the counter."""
		488	+ with (
		489	+ patch("motor.node_manager.core.daemon.HeartbeatManager") as mock_hb_cls,
		490	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		491	+ ):
		492	+ mock_hb = mock_hb_cls.return_value
		493	+ mock_hb.endpoints_generation.return_value = 0
		494	+ mock_hb.is_within_grace_period.return_value = False
		495	+ mock_hb.has_abnormal_endpoints.return_value = True
		496	+
		497	+ daemon._last_endpoints_generation = 0
		498	+ daemon._suicide_abnormal_count = 4
		499	+ mock_hb.has_abnormal_endpoints.return_value = False
		500	+ daemon._check_suicide_condition()
		501	+ assert daemon._suicide_abnormal_count == 0
		502	+
		503	+ daemon._suicide_abnormal_count = 4
		504	+ mock_hb.is_within_grace_period.return_value = True
		505	+ mock_hb.has_abnormal_endpoints.return_value = True
		506	+ daemon._check_suicide_condition()
		507	+ assert daemon._suicide_abnormal_count == 0
		508	+
		509	+ daemon._suicide_abnormal_count = 4
		510	+ mock_hb.endpoints_generation.return_value = 1
		511	+ daemon._check_suicide_condition()
		512	+ assert daemon._suicide_abnormal_count == 0
		513	+
		514	+ assert not daemon.should_suicide()
		515	+
		516	+
		517	+def test_freeze_suspends_counting_and_expires(daemon):
		518	+ """A freeze window suspends counting; deadline expiry or abort resumes it."""
		519	+ with (
		520	+ patch("motor.node_manager.core.daemon.HeartbeatManager") as mock_hb_cls,
		521	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		522	+ ):
		523	+ mock_hb = mock_hb_cls.return_value
		524	+ mock_hb.endpoints_generation.return_value = 0
		525	+ mock_hb.is_within_grace_period.return_value = False
		526	+ mock_hb.has_abnormal_endpoints.return_value = True
		527	+
		528	+ daemon._last_endpoints_generation = 0
		529	+ daemon.freeze_suicide(60)
		530	+ daemon._suicide_abnormal_count = 4
		531	+ daemon._check_suicide_condition()
		532	+ assert daemon._suicide_abnormal_count == 0
		533	+ assert not daemon.should_suicide()
		534	+
		535	+ with patch("motor.node_manager.core.daemon.time.monotonic", return_value=1100.0):
		536	+ assert not daemon.is_suicide_frozen()
		537	+
		538	+ daemon.unfreeze_suicide()
		539	+ assert not daemon.is_suicide_frozen()
		540	+
		541	+
		542	+def test_engine_death_reported_and_deduped_by_pid(daemon):
		543	+ """A dead engine PID freezes suicide and reports once; a fresh PID reports again."""
		544	+ with (
		545	+ patch("motor.node_manager.core.daemon.ControllerApiClient") as mock_client_cls,
		546	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		547	+ ):
		548	+ mock_client_cls.report_software_fault.return_value = True
		549	+ daemon._config.fault_tolerance_config.engine_restart_wait_timeout_sec = 60.0
		550	+ daemon._config.api_config.pod_ip = "10.0.0.1"
		551	+
		552	+ daemon._handle_engine_deaths([(12345, 0)])
		553	+ daemon._handle_engine_deaths([(12345, 0)])
		554	+ daemon._handle_engine_deaths([(54321, 0)])
		555	+
		556	+ assert daemon.is_suicide_frozen()
		557	+ assert mock_client_cls.report_software_fault.call_count == 2
		558	+ fault = mock_client_cls.report_software_fault.call_args_list[0][0][0]
		559	+ assert fault["engine_id"] == 0
		560	+ assert fault["engine_status"] == 1
		561	+ assert fault["exception_type"] == "EngineDeadError"
		562	+ assert fault["pod_ip"] == "10.0.0.1"
		563	+
		564	+
		565	+def test_engine_death_report_failure_retried_without_freeze(daemon):
		566	+ """A failed report is neither deduped nor frozen: the next round retries and
		567	+ the container-restart fallback stays live.
		568	+ """
		569	+ with (
		570	+ patch("motor.node_manager.core.daemon.ControllerApiClient") as mock_client_cls,
		571	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		572	+ ):
		573	+ mock_client_cls.report_software_fault.side_effect = RuntimeError("controller down")
		574	+ daemon._handle_engine_deaths([(12345, 0)])
		575	+ assert 12345 not in daemon._reported_dead_pids
		576	+ assert not daemon.is_suicide_frozen()
		577	+
		578	+ mock_client_cls.report_software_fault.side_effect = None
		579	+ mock_client_cls.report_software_fault.return_value = True
		580	+ daemon._handle_engine_deaths([(12345, 0)])
		581	+ assert 12345 in daemon._reported_dead_pids
		582	+ assert daemon.is_suicide_frozen()
		583	+ assert mock_client_cls.report_software_fault.call_count == 2
		584	+
		585	+
		586	+def test_engine_death_reported_without_freeze_when_relaunch_disabled(daemon):
		587	+ """With enable_engine_relaunch off the death is still reported, but the
		588	+ suicide arbitration is not frozen — the container-restart fallback (k8s)
		589	+ stays live instead of waiting out the freeze window.
		590	+ """
		591	+ with (
		592	+ patch("motor.node_manager.core.daemon.ControllerApiClient") as mock_client_cls,
		593	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		594	+ ):
		595	+ mock_client_cls.report_software_fault.return_value = True
		596	+ daemon._config.fault_tolerance_config.enable_engine_relaunch = False
		597	+ daemon._config.api_config.pod_ip = "10.0.0.1"
		598	+
		599	+ daemon._handle_engine_deaths([(12345, 0)])
		600	+
		601	+ assert mock_client_cls.report_software_fault.call_count == 1
		602	+ assert 12345 in daemon._reported_dead_pids
		603	+ assert not daemon.is_suicide_frozen()
		604	+
		605	+
		606	+def test_abnormal_cold_start_endpoint_not_reported(daemon):
		607	+ """An ABNORMAL endpoint that was never NORMAL is still loading — no death report."""
		608	+ with (
		609	+ patch("motor.node_manager.core.daemon.HeartbeatManager") as mock_hb_cls,
		610	+ patch("motor.node_manager.core.daemon.ControllerApiClient") as mock_client_cls,
		611	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		612	+ ):
		613	+ mock_hb = mock_hb_cls.return_value
		614	+ mock_hb.endpoints_generation.return_value = 0
		615	+ mock_hb.is_within_grace_period.return_value = False
		616	+ mock_hb.has_abnormal_endpoints.return_value = True
		617	+ mock_hb.abnormal_endpoint_ids.return_value = [0, 1]
		618	+ mock_hb.normal_endpoint_ids.return_value = []
		619	+ mock_client_cls.report_software_fault.return_value = True
		620	+
		621	+ daemon._last_endpoints_generation = 0
		622	+ for _ in range(5):
		623	+ daemon._check_suicide_condition()
		624	+
		625	+ assert mock_client_cls.report_software_fault.call_count == 0
		626	+
		627	+
		628	+def test_abnormal_after_normal_reported_until_recovery(daemon):
		629	+ """Once an endpoint turned NORMAL, its ABNORMAL is reported — once, until recovery."""
		630	+ with (
		631	+ patch("motor.node_manager.core.daemon.HeartbeatManager") as mock_hb_cls,
		632	+ patch("motor.node_manager.core.daemon.ControllerApiClient") as mock_client_cls,
		633	+ patch("motor.node_manager.core.daemon.time.monotonic", return_value=1000.0),
		634	+ ):
		635	+ mock_hb = mock_hb_cls.return_value
		636	+ mock_hb.endpoints_generation.return_value = 0
		637	+ mock_hb.is_within_grace_period.return_value = False
		638	+ mock_client_cls.report_software_fault.return_value = True
		639	+
		640	+ mock_hb.has_abnormal_endpoints.return_value = False
		641	+ mock_hb.normal_endpoint_ids.return_value = [0]
		642	+ daemon._last_endpoints_generation = 0
		643	+ daemon._check_suicide_condition()
		644	+
		645	+ mock_hb.has_abnormal_endpoints.return_value = True
		646	+ mock_hb.abnormal_endpoint_ids.return_value = [0]
		647	+ mock_hb.normal_endpoint_ids.return_value = []
		648	+ daemon._check_suicide_condition()
		649	+ daemon._check_suicide_condition()
		650	+ assert mock_client_cls.report_software_fault.call_count == 1
		651	+ assert daemon.is_suicide_frozen()
		652	+
		653	+ daemon.unfreeze_suicide()
		654	+ mock_hb.has_abnormal_endpoints.return_value = False
		655	+ daemon._check_suicide_condition()
		656	+ assert daemon._reported_abnormal_ep_ids == set()
		657	+ mock_hb.has_abnormal_endpoints.return_value = True
		658	+ daemon._check_suicide_condition()
		659	+ assert mock_client_cls.report_software_fault.call_count == 2

Mtests/node_manager/core/test_fault_reporter.py+89-71

	@@ -229,28 +229,28 @@ def test_update_config_no_restart_on_poll_interval_change(reporter, config, endp
229		229
230	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	230	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
231	def test_process_healthy_updates_known_no_report(mock_report, reporter):	231	def test_process_healthy_updates_known_no_report(mock_report, reporter):
232	- known = {}	232	+ reporter._known_statuses = {}
233	- reporter._process_engine_status(0, {"id": 0, "status": "healthy"}, known)	233	+ reporter._process_engine_status(0, {"id": 0, "status": "healthy"})
234	mock_report.assert_not_called()	234	mock_report.assert_not_called()
235	- assert known == {0: "healthy"}	235	+ assert reporter._known_statuses == {0: "healthy"}
236		236
237		237
238	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	238	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
239	def test_process_unhealthy_with_fault_info(mock_report, reporter):	239	def test_process_unhealthy_with_fault_info(mock_report, reporter):
240	- known = {}	240	+ reporter._known_statuses = {}
241	- reporter._process_engine_status(0, {"id": 0, "status": "unhealthy", "fault_info": "RuntimeError"}, known)	241	+ reporter._process_engine_status(0, {"id": 0, "status": "unhealthy", "fault_info": "RuntimeError"})
242	mock_report.assert_called_once()	242	mock_report.assert_called_once()
243	called = mock_report.call_args[0][0]	243	called = mock_report.call_args[0][0]
244	assert called["engine_id"] == 0	244	assert called["engine_id"] == 0
245	assert called["engine_status"] == 2	245	assert called["engine_status"] == 2
246	assert called["exception_type"] == "RuntimeError"	246	assert called["exception_type"] == "RuntimeError"
247	- assert known == {0: "unhealthy"}	247	+ assert reporter._known_statuses == {0: "unhealthy"}
248		248
249		249
250	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	250	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
251	def test_process_unhealthy_without_fault_info(mock_report, reporter):	251	def test_process_unhealthy_without_fault_info(mock_report, reporter):
252	- known = {}	252	+ reporter._known_statuses = {}
253	- reporter._process_engine_status(0, {"id": 0, "status": "unhealthy"}, known)	253	+ reporter._process_engine_status(0, {"id": 0, "status": "unhealthy"})
254	mock_report.assert_called_once()	254	mock_report.assert_called_once()
255	called = mock_report.call_args[0][0]	255	called = mock_report.call_args[0][0]
256	assert called["exception_type"] == "EngineUnhealthyError"	256	assert called["exception_type"] == "EngineUnhealthyError"
	@@ -258,40 +258,40 @@ def test_process_unhealthy_without_fault_info(mock_report, reporter):
258		258
259	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	259	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
260	def test_process_dead(mock_report, reporter):	260	def test_process_dead(mock_report, reporter):
261	- known = {}	261	+ reporter._known_statuses = {}
262	- reporter._process_engine_status(0, {"id": 0, "status": "dead"}, known)	262	+ reporter._process_engine_status(0, {"id": 0, "status": "dead"})
263	mock_report.assert_called_once()	263	mock_report.assert_called_once()
264	called = mock_report.call_args[0][0]	264	called = mock_report.call_args[0][0]
265	assert called["engine_id"] == 0	265	assert called["engine_id"] == 0
266	assert called["engine_status"] == 1	266	assert called["engine_status"] == 1
267	assert called["exception_type"] == "EngineDeadError"	267	assert called["exception_type"] == "EngineDeadError"
268	- assert known == {0: "dead"}	268	+ assert reporter._known_statuses == {0: "dead"}
269		269
270		270
271	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	271	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
272	def test_process_dedup_same_status(mock_report, reporter):	272	def test_process_dedup_same_status(mock_report, reporter):
273	- known = {0: "dead"}	273	+ reporter._known_statuses = {0: "dead"}
274	- reporter._process_engine_status(0, {"id": 0, "status": "dead"}, known)	274	+ reporter._process_engine_status(0, {"id": 0, "status": "dead"})
275	mock_report.assert_not_called()	275	mock_report.assert_not_called()
276		276
277		277
278	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	278	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
279	def test_process_unknown_status(mock_report, reporter):	279	def test_process_unknown_status(mock_report, reporter):
280	- known = {}	280	+ reporter._known_statuses = {}
281	- reporter._process_engine_status(0, {"id": 0, "status": "weird"}, known)	281	+ reporter._process_engine_status(0, {"id": 0, "status": "weird"})
282	mock_report.assert_not_called()	282	mock_report.assert_not_called()
283	- assert known == {}	283	+ assert reporter._known_statuses == {}
284		284
285		285
286	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	286	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
287	def test_process_recovered_then_faulted_again(mock_report, reporter):	287	def test_process_recovered_then_faulted_again(mock_report, reporter):
288	"""After a healthy recovery resets the known status, a new fault is reported."""	288	"""After a healthy recovery resets the known status, a new fault is reported."""
289	- known = {0: "unhealthy"}	289	+ reporter._known_statuses = {0: "unhealthy"}
290	- reporter._process_engine_status(0, {"id": 0, "status": "healthy"}, known)	290	+ reporter._process_engine_status(0, {"id": 0, "status": "healthy"})
291	mock_report.assert_not_called()	291	mock_report.assert_not_called()
292	- reporter._process_engine_status(0, {"id": 0, "status": "unhealthy"}, known)	292	+ reporter._process_engine_status(0, {"id": 0, "status": "unhealthy"})
293	mock_report.assert_called_once()	293	mock_report.assert_called_once()
294	- assert known == {0: "unhealthy"}	294	+ assert reporter._known_statuses == {0: "unhealthy"}
295		295
296		296
297	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	297	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
	@@ -300,11 +300,11 @@ def test_process_failed_report_not_deduped(mock_report, reporter):
300	NOT be marked as known so it will be retried on the next poll.	300	NOT be marked as known so it will be retried on the next poll.
301	"""	301	"""
302	mock_report.return_value = False	302	mock_report.return_value = False
303	- known: dict[int, str] = {}	303	+ reporter._known_statuses = {}
304	- reporter._process_engine_status(0, {"id": 0, "status": "dead"}, known)	304	+ reporter._process_engine_status(0, {"id": 0, "status": "dead"})
305		305
306	mock_report.assert_called_once()	306	mock_report.assert_called_once()
307	- assert 0 not in known	307	+ assert 0 not in reporter._known_statuses
308		308
309		309
310	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	310	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
	@@ -313,11 +313,11 @@ def test_process_successful_report_marked_as_known(mock_report, reporter):
313	IS marked as known so subsequent identical polls are deduplicated.	313	IS marked as known so subsequent identical polls are deduplicated.
314	"""	314	"""
315	mock_report.return_value = True	315	mock_report.return_value = True
316	- known: dict[int, str] = {}	316	+ reporter._known_statuses = {}
317	- reporter._process_engine_status(0, {"id": 0, "status": "dead"}, known)	317	+ reporter._process_engine_status(0, {"id": 0, "status": "dead"})
318		318
319	mock_report.assert_called_once()	319	mock_report.assert_called_once()
320	- assert known == {0: "dead"}	320	+ assert reporter._known_statuses == {0: "dead"}
321		321
322		322
323	# -- status polling ------------------------------------------------------------	323	# -- status polling ------------------------------------------------------------
	@@ -341,36 +341,36 @@ def test_query_engine_status_uses_business_port(mock_client_cls, config, endpoin
341	def test_poll_engine_healthy_resets_failures(reporter, endpoints):	341	def test_poll_engine_healthy_resets_failures(reporter, endpoints):
342	"""A successful poll clears the consecutive-failure counter and reports nothing."""	342	"""A successful poll clears the consecutive-failure counter and reports nothing."""
343	ep = endpoints[0]	343	ep = endpoints[0]
344	- known: dict[int, str] = {}	344	+ reporter._known_statuses = {}
345	- failures: dict[int, int] = {0: 2}	345	+ reporter._consecutive_failures = {0: 2}
346		346
347	with patch.object(	347	with patch.object(
348	reporter,	348	reporter,
349	"_query_engine_status",	349	"_query_engine_status",
350	return_value={"engines": [{"id": 0, "status": "healthy"}]},	350	return_value={"engines": [{"id": 0, "status": "healthy"}]},
351	):	351	):
352	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	352	+ reporter._poll_engine(ep)
353		353
354	- assert failures == {0: 0}	354	+ assert reporter._consecutive_failures == {0: 0}
355	- assert known == {0: "healthy"}	355	+ assert reporter._known_statuses == {0: "healthy"}
356		356
357		357
358	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	358	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
359	def test_poll_engine_unhealthy_reports(mock_report, reporter, endpoints):	359	def test_poll_engine_unhealthy_reports(mock_report, reporter, endpoints):
360	ep = endpoints[0]	360	ep = endpoints[0]
361	- known: dict[int, str] = {}	361	+ reporter._known_statuses = {}
362	- failures: dict[int, int] = {}	362	+ reporter._consecutive_failures = {}
363		363
364	with patch.object(	364	with patch.object(
365	reporter,	365	reporter,
366	"_query_engine_status",	366	"_query_engine_status",
367	return_value={"engines": [{"id": 0, "status": "unhealthy", "fault_info": "KeyError"}]},	367	return_value={"engines": [{"id": 0, "status": "unhealthy", "fault_info": "KeyError"}]},
368	):	368	):
369	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	369	+ reporter._poll_engine(ep)
370		370
371	mock_report.assert_called_once()	371	mock_report.assert_called_once()
372	- assert known == {0: "unhealthy"}	372	+ assert reporter._known_statuses == {0: "unhealthy"}
373	- assert failures == {0: 0}	373	+ assert reporter._consecutive_failures == {0: 0}
374		374
375		375
376	def test_poll_failures_below_threshold_no_report(reporter, endpoints):	376	def test_poll_failures_below_threshold_no_report(reporter, endpoints):
	@@ -378,15 +378,15 @@ def test_poll_failures_below_threshold_no_report(reporter, endpoints):
378	config = reporter._config	378	config = reporter._config
379	config.fault_tolerance_config.max_poll_failures = 3	379	config.fault_tolerance_config.max_poll_failures = 3
380	ep = endpoints[0]	380	ep = endpoints[0]
381	- known: dict[int, str] = {}	381	+ reporter._known_statuses = {}
382	- failures: dict[int, int] = {}	382	+ reporter._consecutive_failures = {}
383		383
384	with patch.object(reporter, "_query_engine_status", side_effect=RuntimeError("boom")):	384	with patch.object(reporter, "_query_engine_status", side_effect=RuntimeError("boom")):
385	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	385	+ reporter._poll_engine(ep)
386	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	386	+ reporter._poll_engine(ep)
387		387
388	- assert failures == {0: 2}	388	+ assert reporter._consecutive_failures == {0: 2}
389	- assert known == {}	389	+ assert reporter._known_statuses == {}
390		390
391		391
392	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	392	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
	@@ -395,12 +395,13 @@ def test_poll_failures_reach_threshold_reports_dead(mock_report, reporter, endpo
395	config = reporter._config	395	config = reporter._config
396	config.fault_tolerance_config.max_poll_failures = 3	396	config.fault_tolerance_config.max_poll_failures = 3
397	ep = endpoints[0]	397	ep = endpoints[0]
398	- known: dict[int, str] = {}	398	+ reporter._known_statuses = {}
399	- failures: dict[int, int] = {}	399	+ reporter._consecutive_failures = {}
		400	+ reporter._first_poll_time = {0: 0} # long ago, startup grace period over
400		401
401	with patch.object(reporter, "_query_engine_status", side_effect=RuntimeError("boom")):	402	with patch.object(reporter, "_query_engine_status", side_effect=RuntimeError("boom")):
402	for _ in range(3):	403	for _ in range(3):
403	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	404	+ reporter._poll_engine(ep)
404		405
405	mock_report.assert_called_once()	406	mock_report.assert_called_once()
406	called = mock_report.call_args[0][0]	407	called = mock_report.call_args[0][0]
	@@ -408,7 +409,7 @@ def test_poll_failures_reach_threshold_reports_dead(mock_report, reporter, endpo
408	assert called["engine_status"] == 1	409	assert called["engine_status"] == 1
409	assert called["exception_type"] == "EngineDeadError"	410	assert called["exception_type"] == "EngineDeadError"
410	assert "unreachable" in called["exception_message"]	411	assert "unreachable" in called["exception_message"]
411	- assert known == {0: "dead"}	412	+ assert reporter._known_statuses == {0: "dead"}
412		413
413		414
414	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")	415	@patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault")
	@@ -417,12 +418,12 @@ def test_poll_failures_dedup_dead(mock_report, reporter, endpoints):
417	config = reporter._config	418	config = reporter._config
418	config.fault_tolerance_config.max_poll_failures = 2	419	config.fault_tolerance_config.max_poll_failures = 2
419	ep = endpoints[0]	420	ep = endpoints[0]
420	- known: dict[int, str] = {0: "dead"} # already reported	421	+ reporter._known_statuses = {0: "dead"} # already reported
421	- failures: dict[int, int] = {}	422	+ reporter._consecutive_failures = {}
422		423
423	with patch.object(reporter, "_query_engine_status", side_effect=RuntimeError("boom")):	424	with patch.object(reporter, "_query_engine_status", side_effect=RuntimeError("boom")):
424	for _ in range(5):	425	for _ in range(5):
425	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	426	+ reporter._poll_engine(ep)
426		427
427	mock_report.assert_not_called()	428	mock_report.assert_not_called()
428		429
	@@ -434,8 +435,8 @@ def test_poll_failures_then_recover(reporter, endpoints):
434	config = reporter._config	435	config = reporter._config
435	config.fault_tolerance_config.max_poll_failures = 3	436	config.fault_tolerance_config.max_poll_failures = 3
436	ep = endpoints[0]	437	ep = endpoints[0]
437	- known: dict[int, str] = {}	438	+ reporter._known_statuses = {}
438	- failures: dict[int, int] = {}	439	+ reporter._consecutive_failures = {}
439		440
440	side_effects = [	441	side_effects = [
441	RuntimeError("boom"),	442	RuntimeError("boom"),
	@@ -445,11 +446,11 @@ def test_poll_failures_then_recover(reporter, endpoints):
445	]	446	]
446	with patch.object(reporter, "_query_engine_status", side_effect=side_effects):	447	with patch.object(reporter, "_query_engine_status", side_effect=side_effects):
447	for _ in range(4):	448	for _ in range(4):
448	- reporter._poll_engine(ep, known, failures, {ep.id: 0})	449	+ reporter._poll_engine(ep)
449		450
450	# 2 failures -> success (reset) -> 1 failure	451	# 2 failures -> success (reset) -> 1 failure
451	- assert failures == {0: 1}	452	+ assert reporter._consecutive_failures == {0: 1}
452	- assert known == {0: "healthy"}	453	+ assert reporter._known_statuses == {0: "healthy"}
453		454
454		455
455	# -- main loop -----------------------------------------------------------------	456	# -- main loop -----------------------------------------------------------------
	@@ -464,7 +465,7 @@ def test_main_loop_polls_all_endpoints_then_stops(reporter, endpoints):
464		465
465	polled: list[int] = []	466	polled: list[int] = []
466		467
467	- def fake_poll(ep, known, failures, first_poll_time):	468	+ def fake_poll(ep):
468	polled.append(ep.id)	469	polled.append(ep.id)
469	r._stop_event.set() # stop after the first full round	470	r._stop_event.set() # stop after the first full round
470		471
	@@ -505,52 +506,52 @@ def test_main_loop_reports_via_sentinel(mock_report, config, endpoints):
505	def test_poll_engine_malformed_payload_does_not_raise(reporter, endpoints):	506	def test_poll_engine_malformed_payload_does_not_raise(reporter, endpoints):
506	"""A malformed FT status payload must not kill the polling thread."""	507	"""A malformed FT status payload must not kill the polling thread."""
507	ep = endpoints[0]	508	ep = endpoints[0]
508	- known: dict[int, str] = {}	509	+ reporter._known_statuses = {}
509	- failures: dict[int, int] = {}	510	+ reporter._consecutive_failures = {}
510		511
511	for bad_payload in ([], None, "ok", {"engines": [{"id": 0}]}, {"engines": ["x"]}):	512	for bad_payload in ([], None, "ok", {"engines": [{"id": 0}]}, {"engines": ["x"]}):
512	with patch.object(reporter, "_query_engine_status", return_value=bad_payload):	513	with patch.object(reporter, "_query_engine_status", return_value=bad_payload):
513	- reporter._poll_engine(ep, known, failures, {ep.id: 0}) # must not raise	514	+ reporter._poll_engine(ep) # must not raise
514		515
515		516
516	def test_process_engine_status_uses_endpoint_id_key(reporter):	517	def test_process_engine_status_uses_endpoint_id_key(reporter):
517	"""Dedup keys are endpoint ids, not payload ids — multi-endpoint safe."""	518	"""Dedup keys are endpoint ids, not payload ids — multi-endpoint safe."""
518	- known: dict[int, str] = {}	519	+ reporter._known_statuses = {}
519	# endpoint 0 dead, endpoint 1 healthy: payload ids collide on 0, keys must not.	520	# endpoint 0 dead, endpoint 1 healthy: payload ids collide on 0, keys must not.
520	with patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault") as mock_report:	521	with patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault") as mock_report:
521	mock_report.return_value = True	522	mock_report.return_value = True
522	- reporter._process_engine_status(0, {"id": 0, "status": "dead"}, known)	523	+ reporter._process_engine_status(0, {"id": 0, "status": "dead"})
523	- reporter._process_engine_status(1, {"id": 0, "status": "healthy"}, known)	524	+ reporter._process_engine_status(1, {"id": 0, "status": "healthy"})
524	- reporter._process_engine_status(0, {"id": 0, "status": "dead"}, known)	525	+ reporter._process_engine_status(0, {"id": 0, "status": "dead"})
525		526
526	- assert known == {0: "dead", 1: "healthy"}	527	+ assert reporter._known_statuses == {0: "dead", 1: "healthy"}
527	mock_report.assert_called_once() # dedup: second dead report suppressed	528	mock_report.assert_called_once() # dedup: second dead report suppressed
528		529
529		530
530	def test_report_unreachable_dead_within_grace_period_not_reported(reporter, endpoints):	531	def test_report_unreachable_dead_within_grace_period_not_reported(reporter, endpoints):
531	"""Poll failures during engine startup (model load) are not reported dead."""	532	"""Poll failures during engine startup (model load) are not reported dead."""
532	ep = endpoints[0]	533	ep = endpoints[0]
533	- known: dict[int, str] = {}	534	+ reporter._known_statuses = {}
534	- first_poll_time = {ep.id: __import__("time").time()}	535	+ reporter._first_poll_time = {ep.id: __import__("time").time()}
535		536
536	with patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault") as mock_report:	537	with patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault") as mock_report:
537	- reporter._report_unreachable_dead(ep, 3, known, first_poll_time)	538	+ reporter._report_unreachable_dead(ep, 3)
538		539
539	mock_report.assert_not_called()	540	mock_report.assert_not_called()
540	- assert known == {}	541	+ assert reporter._known_statuses == {}
541		542
542		543
543	def test_report_unreachable_dead_after_grace_period_reported(reporter, endpoints):	544	def test_report_unreachable_dead_after_grace_period_reported(reporter, endpoints):
544	ep = endpoints[0]	545	ep = endpoints[0]
545	- known: dict[int, str] = {}	546	+ reporter._known_statuses = {}
546	- first_poll_time = {ep.id: 0} # long ago, grace period over	547	+ reporter._first_poll_time = {ep.id: 0} # long ago, grace period over
547		548
548	with patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault") as mock_report:	549	with patch("motor.node_manager.core.fault_reporter.ControllerApiClient.report_software_fault") as mock_report:
549	mock_report.return_value = True	550	mock_report.return_value = True
550	- reporter._report_unreachable_dead(ep, 3, known, first_poll_time)	551	+ reporter._report_unreachable_dead(ep, 3)
551		552
552	mock_report.assert_called_once()	553	mock_report.assert_called_once()
553	- assert known == {0: "dead"}	554	+ assert reporter._known_statuses == {0: "dead"}
554		555
555		556
556	def test_engine_ft_enabled_int_value(tmp_path):	557	def test_engine_ft_enabled_int_value(tmp_path):
	@@ -569,3 +570,20 @@ def test_engine_ft_enabled_ignores_non_engine_sections(tmp_path):
569	{"motor_deploy_config": {"engine_config": {"enable_fault_tolerance": True}}},	570	{"motor_deploy_config": {"engine_config": {"enable_fault_tolerance": True}}},
570	)	571	)
571	assert _engine_ft_enabled(path) is False	572	assert _engine_ft_enabled(path) is False
		573	+
		574	+
		575	+def test_pause_suspends_and_resume_resets_state(reporter, endpoints):
		576	+ """pause() suspends polling; resume() clears all per-endpoint poll state."""
		577	+ ep = endpoints[0]
		578	+ reporter._first_poll_time[ep.id] = 100.0
		579	+ reporter._consecutive_failures[ep.id] = 2
		580	+ reporter._known_statuses[ep.id] = "dead"
		581	+
		582	+ reporter.pause()
		583	+ assert reporter._pause_event.is_set()
		584	+
		585	+ reporter.resume()
		586	+ assert not reporter._pause_event.is_set()
		587	+ assert reporter._first_poll_time == {}
		588	+ assert reporter._consecutive_failures == {}
		589	+ assert reporter._known_statuses == {}

Mtests/node_manager/core/test_heartbeat_manager.py+102-293

	@@ -28,14 +28,14 @@ mock_config.api_config = MagicMock()
28	with patch('motor.config.node_manager.NodeManagerConfig.from_json', return_value=mock_config):	28	with patch('motor.config.node_manager.NodeManagerConfig.from_json', return_value=mock_config):
29	from motor.common.resources.endpoint import Endpoint, EndpointStatus	29	from motor.common.resources.endpoint import Endpoint, EndpointStatus
30	from motor.common.resources.http_msg_spec import StartCmdMsg	30	from motor.common.resources.http_msg_spec import StartCmdMsg
31	- from motor.node_manager.core.engine_manager import EngineManager	31	+ from motor.node_manager.core.register_manager import RegisterManager
32	from motor.node_manager.core.heartbeat_manager import HeartbeatManager	32	from motor.node_manager.core.heartbeat_manager import HeartbeatManager
33	from motor.config.node_manager import NodeManagerConfig	33	from motor.config.node_manager import NodeManagerConfig
34		34
35		35
36	-def _clear_engine_manager_singleton() -> None:	36	+def _clear_register_manager_singleton() -> None:
37	- if hasattr(EngineManager, "_instances") and EngineManager in EngineManager._instances:	37	+ if hasattr(RegisterManager, "_instances") and RegisterManager in RegisterManager._instances:
38	- del EngineManager._instances[EngineManager]	38	+ del RegisterManager._instances[RegisterManager]
39		39
40		40
41	class TestHeartBeatManager:	41	class TestHeartBeatManager:
	@@ -47,7 +47,7 @@ class TestHeartBeatManager:
47	with (	47	with (
48	patch('motor.config.node_manager.safe_open') as mock_safe_open,	48	patch('motor.config.node_manager.safe_open') as mock_safe_open,
49	patch('threading.Thread') as mock_thread_class,	49	patch('threading.Thread') as mock_thread_class,
50	- patch('motor.node_manager.core.heartbeat_manager.EngineManager') as mock_engine_manager_cls,	50	+ patch('motor.node_manager.core.heartbeat_manager.RegisterManager') as mock_register_manager_cls,
51	patch.dict(	51	patch.dict(
52	'os.environ',	52	'os.environ',
53	{	53	{
	@@ -61,10 +61,10 @@ class TestHeartBeatManager:
61	mock_safe_open.side_effect = create_config_mock(config_data)	61	mock_safe_open.side_effect = create_config_mock(config_data)
62	mock_thread = MagicMock()	62	mock_thread = MagicMock()
63	mock_thread_class.return_value = mock_thread	63	mock_thread_class.return_value = mock_thread
64	- mock_engine_manager = MagicMock()	64	+ mock_register_manager = MagicMock()
65	- mock_engine_manager.is_engine_checkpoint_done.return_value = True	65	+ mock_register_manager.is_engine_checkpoint_done.return_value = True
66	- mock_engine_manager_cls.return_value = mock_engine_manager	66	+ mock_register_manager_cls.return_value = mock_register_manager
67	- _clear_engine_manager_singleton()	67	+ _clear_register_manager_singleton()
68	# clear HeartBeatManager instance (HeartbeatManager is still singleton)	68	# clear HeartBeatManager instance (HeartbeatManager is still singleton)
69	if hasattr(HeartbeatManager, '_instances') and HeartbeatManager in HeartbeatManager._instances:	69	if hasattr(HeartbeatManager, '_instances') and HeartbeatManager in HeartbeatManager._instances:
70	try:	70	try:
	@@ -160,7 +160,7 @@ class TestHeartBeatManager:
160		160
161	assert heart_beat_manager._endpoints_generation == before + 1	161	assert heart_beat_manager._endpoints_generation == before + 1
162		162
163	- @patch('motor.node_manager.core.heartbeat_manager.Daemon')	163	+ @patch('motor.node_manager.core.daemon.Daemon')
164	def test_engine_metrics_targets_exclude_headless(self, mock_daemon, heart_beat_manager):	164	def test_engine_metrics_targets_exclude_headless(self, mock_daemon, heart_beat_manager):
165	routable = Endpoint(id=1, ip="10.0.0.1", business_port="8001", mgmt_port="9001")	165	routable = Endpoint(id=1, ip="10.0.0.1", business_port="8001", mgmt_port="9001")
166	headless = Endpoint(	166	headless = Endpoint(
	@@ -179,7 +179,7 @@ class TestHeartBeatManager:
179	assert targets == ["https://10.0.0.1:8001/metrics"]	179	assert targets == ["https://10.0.0.1:8001/metrics"]
180	mock_daemon.return_value.get_engine_metrics_target.assert_called_once_with(routable)	180	mock_daemon.return_value.get_engine_metrics_target.assert_called_once_with(routable)
181		181
182	- @patch('motor.node_manager.core.heartbeat_manager.Daemon')	182	+ @patch('motor.node_manager.core.daemon.Daemon')
183	def test_refresh_native_engine_status_success(self, mock_daemon, heart_beat_manager, sample_endpoints):	183	def test_refresh_native_engine_status_success(self, mock_daemon, heart_beat_manager, sample_endpoints):
184	"""READY native runtimes map to normal endpoint status."""	184	"""READY native runtimes map to normal endpoint status."""
185	from motor.node_manager.core.services.native_engine.models import RuntimeState	185	from motor.node_manager.core.services.native_engine.models import RuntimeState
	@@ -196,7 +196,7 @@ class TestHeartBeatManager:
196	assert heart_beat_manager._endpoints[0].status == EndpointStatus.NORMAL	196	assert heart_beat_manager._endpoints[0].status == EndpointStatus.NORMAL
197	assert heart_beat_manager._endpoints[1].status == EndpointStatus.NORMAL	197	assert heart_beat_manager._endpoints[1].status == EndpointStatus.NORMAL
198		198
199	- @patch('motor.node_manager.core.heartbeat_manager.Daemon')	199	+ @patch('motor.node_manager.core.daemon.Daemon')
200	def test_refresh_native_engine_status_keeps_initial_while_loading(self, mock_daemon, heart_beat_manager):	200	def test_refresh_native_engine_status_keeps_initial_while_loading(self, mock_daemon, heart_beat_manager):
201	from motor.node_manager.core.services.native_engine.models import RuntimeState	201	from motor.node_manager.core.services.native_engine.models import RuntimeState
202		202
	@@ -215,7 +215,7 @@ class TestHeartBeatManager:
215		215
216	assert heart_beat_manager._endpoints[0].status == EndpointStatus.INITIAL	216	assert heart_beat_manager._endpoints[0].status == EndpointStatus.INITIAL
217		217
218	- @patch('motor.node_manager.core.heartbeat_manager.Daemon')	218	+ @patch('motor.node_manager.core.daemon.Daemon')
219	def test_refresh_headless_process_liveness_reports_wait2start(self, mock_daemon, heart_beat_manager):	219	def test_refresh_headless_process_liveness_reports_wait2start(self, mock_daemon, heart_beat_manager):
220	from motor.node_manager.core.services.native_engine.models import RuntimeState	220	from motor.node_manager.core.services.native_engine.models import RuntimeState
221		221
	@@ -235,7 +235,7 @@ class TestHeartBeatManager:
235		235
236	assert heart_beat_manager._endpoints[0].status == EndpointStatus.WAIT2START	236	assert heart_beat_manager._endpoints[0].status == EndpointStatus.WAIT2START
237		237
238	- @patch('motor.node_manager.core.heartbeat_manager.Daemon')	238	+ @patch('motor.node_manager.core.daemon.Daemon')
239	def test_refresh_native_engine_status_discards_stale_probe_write_back(	239	def test_refresh_native_engine_status_discards_stale_probe_write_back(
240	self, mock_daemon, heart_beat_manager, sample_start_cmd_msg	240	self, mock_daemon, heart_beat_manager, sample_start_cmd_msg
241	):	241	):
	@@ -420,34 +420,34 @@ class TestHeartBeatManager:
420	heart_beat_manager.start()	420	heart_beat_manager.start()
421	assert heart_beat_manager._thread_started is True	421	assert heart_beat_manager._thread_started is True
422		422
423	- @patch('motor.node_manager.core.heartbeat_manager.EngineManager')	423	+ @patch('motor.node_manager.core.heartbeat_manager.RegisterManager')
424	- def test_reregister_success(self, mock_engine_manager_class, heart_beat_manager):	424	+ def test_reregister_success(self, mock_register_manager_class, heart_beat_manager):
425	"""test _reregister success"""	425	"""test _reregister success"""
426	- mock_engine_manager = MagicMock()	426	+ mock_register_manager = MagicMock()
427	- mock_engine_manager.post_reregister_msg.return_value = True	427	+ mock_register_manager.post_reregister_msg.return_value = True
428	- mock_engine_manager_class.return_value = mock_engine_manager	428	+ mock_register_manager_class.return_value = mock_register_manager
429		429
430	heart_beat_manager._reregister()	430	heart_beat_manager._reregister()
431		431
432	- mock_engine_manager.post_reregister_msg.assert_called_once()	432	+ mock_register_manager.post_reregister_msg.assert_called_once()
433		433
434	- @patch('motor.node_manager.core.heartbeat_manager.EngineManager')	434	+ @patch('motor.node_manager.core.heartbeat_manager.RegisterManager')
435	- def test_reregister_failure(self, mock_engine_manager_class, heart_beat_manager):	435	+ def test_reregister_failure(self, mock_register_manager_class, heart_beat_manager):
436	"""test _reregister failure"""	436	"""test _reregister failure"""
437	- mock_engine_manager = MagicMock()	437	+ mock_register_manager = MagicMock()
438	- mock_engine_manager.post_reregister_msg.return_value = False	438	+ mock_register_manager.post_reregister_msg.return_value = False
439	- mock_engine_manager_class.return_value = mock_engine_manager	439	+ mock_register_manager_class.return_value = mock_register_manager
440		440
441	heart_beat_manager._reregister()	441	heart_beat_manager._reregister()
442		442
443	- mock_engine_manager.post_reregister_msg.assert_called_once()	443	+ mock_register_manager.post_reregister_msg.assert_called_once()
444		444
445	@patch('motor.node_manager.core.heartbeat_manager.threading.Thread')	445	@patch('motor.node_manager.core.heartbeat_manager.threading.Thread')
446	@patch('motor.node_manager.core.heartbeat_manager.time.sleep')	446	@patch('motor.node_manager.core.heartbeat_manager.time.sleep')
447	@patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')	447	@patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
448	- @patch('motor.node_manager.core.heartbeat_manager.EngineManager')	448	+ @patch('motor.node_manager.core.heartbeat_manager.RegisterManager')
449	def test_reregister_triggered_on_503(	449	def test_reregister_triggered_on_503(
450	- self, mock_engine_manager_class, mock_report_heartbeat, mock_sleep, mock_thread_class, heart_beat_manager	450	+ self, mock_register_manager_class, mock_report_heartbeat, mock_sleep, mock_thread_class, heart_beat_manager
451	):	451	):
452	"""test that reregister is triggered when 503 error occurs"""	452	"""test that reregister is triggered when 503 error occurs"""
453	call_count = {"count": 0}	453	call_count = {"count": 0}
	@@ -460,10 +460,10 @@ class TestHeartBeatManager:
460	# Mock report_heartbeat to raise 503 error	460	# Mock report_heartbeat to raise 503 error
461	mock_report_heartbeat.side_effect = Exception("503 Service Unavailable")	461	mock_report_heartbeat.side_effect = Exception("503 Service Unavailable")
462		462
463	- mock_engine_manager = MagicMock()	463	+ mock_register_manager = MagicMock()
464	- mock_engine_manager.is_engine_checkpoint_done.return_value = True	464	+ mock_register_manager.is_engine_checkpoint_done.return_value = True
465	- mock_engine_manager.post_reregister_msg.return_value = True	465	+ mock_register_manager.post_reregister_msg.return_value = True
466	- mock_engine_manager_class.return_value = mock_engine_manager	466	+ mock_register_manager_class.return_value = mock_register_manager
467		467
468	mock_reregister_thread = MagicMock()	468	mock_reregister_thread = MagicMock()
469	mock_thread_class.return_value = mock_reregister_thread	469	mock_thread_class.return_value = mock_reregister_thread
	@@ -477,14 +477,14 @@ class TestHeartBeatManager:
477		477
478	heart_beat_manager._report_heartbeat_loop()	478	heart_beat_manager._report_heartbeat_loop()
479		479
480	- # Verify that reregister was called (via EngineManager)	480	+ # Verify that reregister was called (via RegisterManager)
481	- mock_engine_manager.post_reregister_msg.assert_called()	481	+ mock_register_manager.post_reregister_msg.assert_called()
482		482
483	@patch('motor.node_manager.core.heartbeat_manager.time.sleep')	483	@patch('motor.node_manager.core.heartbeat_manager.time.sleep')
484	@patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')	484	@patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
485	- @patch('motor.node_manager.core.heartbeat_manager.EngineManager')	485	+ @patch('motor.node_manager.core.heartbeat_manager.RegisterManager')
486	def test_reregister_lock_thread_safety(	486	def test_reregister_lock_thread_safety(
487	- self, mock_engine_manager_class, mock_report_heartbeat, mock_sleep, heart_beat_manager	487	+ self, mock_register_manager_class, mock_report_heartbeat, mock_sleep, heart_beat_manager
488	):	488	):
489	"""test that _reregister_lock prevents concurrent reregister attempts"""	489	"""test that _reregister_lock prevents concurrent reregister attempts"""
490	call_count = {"count": 0}	490	call_count = {"count": 0}
	@@ -494,11 +494,11 @@ class TestHeartBeatManager:
494	heart_beat_manager.stop_event.set()	494	heart_beat_manager.stop_event.set()
495	call_count["count"] += 1	495	call_count["count"] += 1
496		496
497	- # Mock EngineManager	497	+ # Mock RegisterManager
498	- mock_engine_manager = MagicMock()	498	+ mock_register_manager = MagicMock()
499	- mock_engine_manager.is_engine_checkpoint_done.return_value = True	499	+ mock_register_manager.is_engine_checkpoint_done.return_value = True
500	- mock_engine_manager.post_reregister_msg.return_value = True	500	+ mock_register_manager.post_reregister_msg.return_value = True
501	- mock_engine_manager_class.return_value = mock_engine_manager	501	+ mock_register_manager_class.return_value = mock_register_manager
502		502
503	# Mock report_heartbeat to raise 503 error	503	# Mock report_heartbeat to raise 503 error
504	mock_report_heartbeat.side_effect = Exception("503 Service Unavailable")	504	mock_report_heartbeat.side_effect = Exception("503 Service Unavailable")
	@@ -531,224 +531,6 @@ class TestHeartBeatManager:
531		531
532	assert heart_beat_manager.stop_event.is_set() is True	532	assert heart_beat_manager.stop_event.is_set() is True
533		533
534	- def test_initial_suicide_flag(self, heart_beat_manager):
535	- """test that suicide flag is initially False"""
536	- assert heart_beat_manager.should_suicide() is False
537	- assert heart_beat_manager._consecutive_abnormal_count == 0
538	-
539	- @patch('motor.node_manager.core.heartbeat_manager.time.sleep')
540	- @patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
541	- def test_consecutive_abnormal_heartbeat_counting(self, mock_report_heartbeat, mock_sleep, heart_beat_manager):
542	- """test that consecutive abnormal heartbeats are counted correctly"""
543	- call_count = {"count": 0}
544	-
545	- def mock_stop_sleep(seconds):
546	- call_count["count"] += 1
547	- if call_count["count"] >= 6: # Run 6 times to test 5 consecutive abnormal
548	- heart_beat_manager.stop_event.set()
549	-
550	- mock_report_heartbeat.return_value = None
551	-
552	- # Set endpoint info with abnormal status
553	- heart_beat_manager._job_name = "test_job"
554	- heart_beat_manager._instance_id = 1
555	- heart_beat_manager.stop_event.clear()
556	-
557	- with heart_beat_manager._endpoint_lock:
558	- heart_beat_manager._endpoints = [
559	- Endpoint(id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.ABNORMAL)
560	- ]
561	-
562	- mock_sleep.side_effect = mock_stop_sleep
563	-
564	- # Run the heartbeat loop
565	- heart_beat_manager._report_heartbeat_loop()
566	-
567	- # After 5 consecutive abnormal heartbeats, suicide flag should be set
568	- assert heart_beat_manager.should_suicide() is True
569	- assert heart_beat_manager._consecutive_abnormal_count >= 5
570	-
571	- @patch('motor.node_manager.core.heartbeat_manager.time.sleep')
572	- @patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
573	- def test_abnormal_count_reset_on_normal_status(self, mock_report_heartbeat, mock_sleep, heart_beat_manager):
574	- """test that abnormal count resets when status returns to normal"""
575	- call_count = {"count": 0}
576	-
577	- def mock_stop_sleep(seconds):
578	- call_count["count"] += 1
579	- # Change status to normal after first iteration
580	- if call_count["count"] == 1:
581	- with heart_beat_manager._endpoint_lock:
582	- if heart_beat_manager._endpoints:
583	- heart_beat_manager._endpoints[0].status = EndpointStatus.NORMAL
584	- if call_count["count"] >= 3:
585	- heart_beat_manager.stop_event.set()
586	-
587	- mock_report_heartbeat.return_value = None
588	-
589	- heart_beat_manager._job_name = "test_job"
590	- heart_beat_manager._instance_id = 1
591	- heart_beat_manager.stop_event.clear()
592	-
593	- # Start with abnormal status
594	- with heart_beat_manager._endpoint_lock:
595	- heart_beat_manager._endpoints = [
596	- Endpoint(id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.ABNORMAL)
597	- ]
598	-
599	- mock_sleep.side_effect = mock_stop_sleep
600	-
601	- heart_beat_manager._report_heartbeat_loop()
602	-
603	- # After status returns to normal, count should be reset
604	- assert heart_beat_manager._consecutive_abnormal_count == 0
605	- assert heart_beat_manager.should_suicide() is False
606	-
607	- def test_update_endpoint_resets_abnormal_count(self, heart_beat_manager, sample_start_cmd_msg):
608	- """test that updating endpoint resets abnormal count and suicide flag"""
609	- # Set abnormal count and suicide flag first
610	- with heart_beat_manager._abnormal_count_lock:
611	- heart_beat_manager._consecutive_abnormal_count = 5
612	- with heart_beat_manager._suicide_lock:
613	- heart_beat_manager._should_suicide = True
614	-
615	- # Update endpoint should reset both
616	- heart_beat_manager.update_endpoint(sample_start_cmd_msg)
617	-
618	- assert heart_beat_manager._consecutive_abnormal_count == 0
619	- assert heart_beat_manager.should_suicide() is False
620	-
621	- @patch('motor.node_manager.core.heartbeat_manager.time.sleep')
622	- @patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
623	- def test_suicide_flag_set_after_five_abnormal_heartbeats(
624	- self, mock_report_heartbeat, mock_sleep, heart_beat_manager
625	- ):
626	- """test that suicide flag is set exactly after 5 consecutive abnormal heartbeats"""
627	- call_count = {"count": 0}
628	-
629	- def mock_stop_sleep(seconds):
630	- call_count["count"] += 1
631	- if call_count["count"] >= 5:
632	- heart_beat_manager.stop_event.set()
633	-
634	- mock_report_heartbeat.return_value = None
635	-
636	- heart_beat_manager._job_name = "test_job"
637	- heart_beat_manager._instance_id = 1
638	- heart_beat_manager.stop_event.clear()
639	-
640	- with heart_beat_manager._endpoint_lock:
641	- heart_beat_manager._endpoints = [
642	- Endpoint(id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.ABNORMAL)
643	- ]
644	-
645	- mock_sleep.side_effect = mock_stop_sleep
646	-
647	- # Initially suicide flag should be False
648	- assert heart_beat_manager.should_suicide() is False
649	-
650	- heart_beat_manager._report_heartbeat_loop()
651	-
652	- # After 5 consecutive abnormal heartbeats, suicide flag should be True
653	- assert heart_beat_manager.should_suicide() is True
654	- assert heart_beat_manager._consecutive_abnormal_count == 5
655	-
656	- @patch('motor.node_manager.core.heartbeat_manager.time.sleep')
657	- @patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
658	- def test_multiple_endpoints_abnormal_triggers_suicide(self, mock_report_heartbeat, mock_sleep, heart_beat_manager):
659	- """test that if any endpoint is abnormal, it counts towards suicide"""
660	- call_count = {"count": 0}
661	-
662	- def mock_stop_sleep(seconds):
663	- call_count["count"] += 1
664	- if call_count["count"] >= 5:
665	- heart_beat_manager.stop_event.set()
666	-
667	- mock_report_heartbeat.return_value = None
668	-
669	- heart_beat_manager._job_name = "test_job"
670	- heart_beat_manager._instance_id = 1
671	- heart_beat_manager.stop_event.clear()
672	-
673	- # Set multiple endpoints, one abnormal
674	- with heart_beat_manager._endpoint_lock:
675	- heart_beat_manager._endpoints = [
676	- Endpoint(
677	- id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.ABNORMAL
678	- ),
679	- Endpoint(id=2, ip="192.168.1.2", business_port="8080", mgmt_port="9090", status=EndpointStatus.NORMAL),
680	- ]
681	-
682	- mock_sleep.side_effect = mock_stop_sleep
683	-
684	- heart_beat_manager._report_heartbeat_loop()
685	-
686	- # Even with one endpoint abnormal, suicide should be triggered after 5 consecutive reports
687	- assert heart_beat_manager.should_suicide() is True
688	-
689	- @patch('motor.node_manager.core.heartbeat_manager.time.sleep')
690	- @patch('motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat')
691	- def test_abnormal_triggers_suicide_when_report_fails(self, mock_report_heartbeat, mock_sleep, heart_beat_manager):
692	- """endpoint stays abnormal but Controller heartbeat report fails should still trigger suicide"""
693	- call_count = {"count": 0}
694	-
695	- def mock_stop_sleep(seconds):
696	- call_count["count"] += 1
697	- if call_count["count"] >= 5:
698	- heart_beat_manager.stop_event.set()
699	-
700	- mock_report_heartbeat.side_effect = Exception("Connection refused")
701	-
702	- heart_beat_manager._job_name = "test_job"
703	- heart_beat_manager._instance_id = 1
704	- heart_beat_manager.stop_event.clear()
705	-
706	- with heart_beat_manager._endpoint_lock:
707	- heart_beat_manager._endpoints = [
708	- Endpoint(id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.ABNORMAL)
709	- ]
710	-
711	- mock_sleep.side_effect = mock_stop_sleep
712	-
713	- heart_beat_manager._report_heartbeat_loop()
714	-
715	- assert heart_beat_manager.should_suicide() is True
716	- assert heart_beat_manager._consecutive_abnormal_count == 5
717	-
718	- @patch('motor.node_manager.core.heartbeat_manager.threading.Thread')
719	- def test_should_suicide_thread_safety(self, mock_thread_class, heart_beat_manager):
720	- """test that should_suicide method is thread-safe"""
721	-
722	- # Set suicide flag
723	- with heart_beat_manager._suicide_lock:
724	- heart_beat_manager._should_suicide = True
725	-
726	- # Verify flag is set
727	- assert heart_beat_manager.should_suicide() is True
728	-
729	- # Test that the lock protects the flag correctly
730	- # We'll test by calling should_suicide multiple times and verifying consistency
731	- results = []
732	- for _ in range(10):
733	- results.append(heart_beat_manager.should_suicide())
734	-
735	- # All calls should get the same result (True)
736	- assert len(results) == 10
737	- assert all(results), f"All results should be True, got {results}"
738	-
739	- # Test concurrent access simulation by checking lock behavior
740	- # Reset flag and test again
741	- with heart_beat_manager._suicide_lock:
742	- heart_beat_manager._should_suicide = False
743	-
744	- results2 = []
745	- for _ in range(10):
746	- results2.append(heart_beat_manager.should_suicide())
747	-
748	- # All calls should get False now
749	- assert len(results2) == 10
750	- assert all(r is False for r in results2), f"All results should be False, got {results2}"
751	-
752	def test_is_started_after_restore_defaults_false(self, heart_beat_manager):	534	def test_is_started_after_restore_defaults_false(self, heart_beat_manager):
753	assert heart_beat_manager.is_started_after_restore() is False	535	assert heart_beat_manager.is_started_after_restore() is False
754		536
	@@ -756,37 +538,37 @@ class TestHeartBeatManager:
756	heart_beat_manager.set_started_after_restore(True)	538	heart_beat_manager.set_started_after_restore(True)
757	assert heart_beat_manager.is_started_after_restore() is True	539	assert heart_beat_manager.is_started_after_restore() is True
758		540
759	- @patch("motor.node_manager.core.heartbeat_manager.EngineManager")	541	+ @patch("motor.node_manager.core.heartbeat_manager.RegisterManager")
760	- def test_register_after_restore_success(self, mock_engine_manager_class, heart_beat_manager):	542	+ def test_register_after_restore_success(self, mock_register_manager_class, heart_beat_manager):
761	- mock_engine_manager = MagicMock()	543	+ mock_register_manager = MagicMock()
762	- mock_engine_manager.post_register_msg.return_value = True	544	+ mock_register_manager.post_register_msg.return_value = True
763	- mock_engine_manager_class.return_value = mock_engine_manager	545	+ mock_register_manager_class.return_value = mock_register_manager
764		546
765	heart_beat_manager._register_after_restore()	547	heart_beat_manager._register_after_restore()
766		548
767	- mock_engine_manager.register_prepare_after_restore.assert_called_once()	549	+ mock_register_manager.register_prepare_after_restore.assert_called_once()
768	- mock_engine_manager.post_register_msg.assert_called_once()	550	+ mock_register_manager.post_register_msg.assert_called_once()
769	assert heart_beat_manager._is_registered_after_restore is True	551	assert heart_beat_manager._is_registered_after_restore is True
770		552
771	- @patch("motor.node_manager.core.heartbeat_manager.EngineManager")	553	+ @patch("motor.node_manager.core.heartbeat_manager.RegisterManager")
772	- def test_register_after_restore_prepare_failure(self, mock_engine_manager_class, heart_beat_manager):	554	+ def test_register_after_restore_prepare_failure(self, mock_register_manager_class, heart_beat_manager):
773	- mock_engine_manager = MagicMock()	555	+ mock_register_manager = MagicMock()
774	- mock_engine_manager.register_prepare_after_restore.side_effect = RuntimeError("metadata missing")	556	+ mock_register_manager.register_prepare_after_restore.side_effect = RuntimeError("metadata missing")
775	- mock_engine_manager_class.return_value = mock_engine_manager	557	+ mock_register_manager_class.return_value = mock_register_manager
776		558
777	heart_beat_manager._register_after_restore()	559	heart_beat_manager._register_after_restore()
778		560
779	- mock_engine_manager.post_register_msg.assert_not_called()	561	+ mock_register_manager.post_register_msg.assert_not_called()
780	assert heart_beat_manager._is_registered_after_restore is False	562	assert heart_beat_manager._is_registered_after_restore is False
781	assert heart_beat_manager._register_after_restore_retry_count == 1	563	assert heart_beat_manager._register_after_restore_retry_count == 1
782		564
783	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=True)	565	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=True)
784	@patch("motor.node_manager.core.heartbeat_manager.time.sleep")	566	@patch("motor.node_manager.core.heartbeat_manager.time.sleep")
785	@patch("motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat")	567	@patch("motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat")
786	- @patch("motor.node_manager.core.heartbeat_manager.EngineManager")	568	+ @patch("motor.node_manager.core.heartbeat_manager.RegisterManager")
787	def test_report_heartbeat_loop_registers_before_reporting(	569	def test_report_heartbeat_loop_registers_before_reporting(
788	self,	570	self,
789	- mock_engine_manager_class,	571	+ mock_register_manager_class,
790	mock_report_heartbeat,	572	mock_report_heartbeat,
791	mock_sleep,	573	mock_sleep,
792	_mock_restored,	574	_mock_restored,
	@@ -800,11 +582,11 @@ class TestHeartBeatManager:
800	if call_count["count"] >= 2:	582	if call_count["count"] >= 2:
801	heart_beat_manager.stop_event.set()	583	heart_beat_manager.stop_event.set()
802		584
803	- mock_engine_manager = MagicMock()	585	+ mock_register_manager = MagicMock()
804	- mock_engine_manager.is_engine_checkpoint_done.return_value = True	586	+ mock_register_manager.is_engine_checkpoint_done.return_value = True
805	- mock_engine_manager.register_prepare_after_restore.return_value = None	587	+ mock_register_manager.register_prepare_after_restore.return_value = None
806	- mock_engine_manager.post_register_msg.return_value = True	588	+ mock_register_manager.post_register_msg.return_value = True
807	- mock_engine_manager_class.return_value = mock_engine_manager	589	+ mock_register_manager_class.return_value = mock_register_manager
808	mock_report_heartbeat.return_value = None	590	mock_report_heartbeat.return_value = None
809	mock_sleep.side_effect = mock_stop_sleep	591	mock_sleep.side_effect = mock_stop_sleep
810		592
	@@ -818,12 +600,12 @@ class TestHeartBeatManager:
818		600
819	heart_beat_manager._report_heartbeat_loop()	601	heart_beat_manager._report_heartbeat_loop()
820		602
821	- mock_engine_manager.register_prepare_after_restore.assert_called_once()	603	+ mock_register_manager.register_prepare_after_restore.assert_called_once()
822	- mock_engine_manager.post_register_msg.assert_called_once()	604	+ mock_register_manager.post_register_msg.assert_called_once()
823	mock_report_heartbeat.assert_called_once()	605	mock_report_heartbeat.assert_called_once()
824		606
825	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=True)	607	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=True)
826	- @patch("motor.node_manager.core.heartbeat_manager.Daemon")	608	+ @patch('motor.node_manager.core.daemon.Daemon')
827	def test_refresh_native_engine_status_keeps_status_before_start_after_restore(	609	def test_refresh_native_engine_status_keeps_status_before_start_after_restore(
828	self, mock_daemon, _mock_restored, heart_beat_manager, sample_endpoints	610	self, mock_daemon, _mock_restored, heart_beat_manager, sample_endpoints
829	):	611	):
	@@ -843,9 +625,9 @@ class TestHeartBeatManager:
843	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=False)	625	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=False)
844	@patch("motor.node_manager.core.heartbeat_manager.time.sleep")	626	@patch("motor.node_manager.core.heartbeat_manager.time.sleep")
845	@patch("motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat")	627	@patch("motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat")
846	- @patch("motor.node_manager.core.heartbeat_manager.EngineManager")	628	+ @patch("motor.node_manager.core.heartbeat_manager.RegisterManager")
847	def test_report_heartbeat_skipped_until_checkpoint_done(	629	def test_report_heartbeat_skipped_until_checkpoint_done(
848	- self, mock_engine_manager_class, mock_report_heartbeat, mock_sleep, _mock_restored, heart_beat_manager	630	+ self, mock_register_manager_class, mock_report_heartbeat, mock_sleep, _mock_restored, heart_beat_manager
849	):	631	):
850	call_count = {"count": 0}	632	call_count = {"count": 0}
851		633
	@@ -854,9 +636,9 @@ class TestHeartBeatManager:
854	if call_count["count"] >= 1:	636	if call_count["count"] >= 1:
855	heart_beat_manager.stop_event.set()	637	heart_beat_manager.stop_event.set()
856		638
857	- mock_engine_manager = MagicMock()	639	+ mock_register_manager = MagicMock()
858	- mock_engine_manager.is_engine_checkpoint_done.return_value = False	640	+ mock_register_manager.is_engine_checkpoint_done.return_value = False
859	- mock_engine_manager_class.return_value = mock_engine_manager	641	+ mock_register_manager_class.return_value = mock_register_manager
860	mock_sleep.side_effect = mock_stop_sleep	642	mock_sleep.side_effect = mock_stop_sleep
861		643
862	heart_beat_manager._job_name = "test_job"	644	heart_beat_manager._job_name = "test_job"
	@@ -870,14 +652,14 @@ class TestHeartBeatManager:
870	heart_beat_manager._report_heartbeat_loop()	652	heart_beat_manager._report_heartbeat_loop()
871		653
872	mock_report_heartbeat.assert_not_called()	654	mock_report_heartbeat.assert_not_called()
873	- mock_engine_manager.is_engine_checkpoint_done.assert_called()	655	+ mock_register_manager.is_engine_checkpoint_done.assert_called()
874		656
875	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=False)	657	@patch("motor.node_manager.core.heartbeat_manager.is_restored_from_host_side_snapshot", return_value=False)
876	@patch("motor.node_manager.core.heartbeat_manager.time.sleep")	658	@patch("motor.node_manager.core.heartbeat_manager.time.sleep")
877	@patch("motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat")	659	@patch("motor.node_manager.core.heartbeat_manager.ControllerApiClient.report_heartbeat")
878	- @patch("motor.node_manager.core.heartbeat_manager.EngineManager")	660	+ @patch("motor.node_manager.core.heartbeat_manager.RegisterManager")
879	def test_report_heartbeat_resumes_after_checkpoint_done(	661	def test_report_heartbeat_resumes_after_checkpoint_done(
880	- self, mock_engine_manager_class, mock_report_heartbeat, mock_sleep, _mock_restored, heart_beat_manager	662	+ self, mock_register_manager_class, mock_report_heartbeat, mock_sleep, _mock_restored, heart_beat_manager
881	):	663	):
882	call_count = {"count": 0}	664	call_count = {"count": 0}
883		665
	@@ -886,9 +668,9 @@ class TestHeartBeatManager:
886	if call_count["count"] >= 1:	668	if call_count["count"] >= 1:
887	heart_beat_manager.stop_event.set()	669	heart_beat_manager.stop_event.set()
888		670
889	- mock_engine_manager = MagicMock()	671	+ mock_register_manager = MagicMock()
890	- mock_engine_manager.is_engine_checkpoint_done.return_value = True	672	+ mock_register_manager.is_engine_checkpoint_done.return_value = True
891	- mock_engine_manager_class.return_value = mock_engine_manager	673	+ mock_register_manager_class.return_value = mock_register_manager
892	mock_report_heartbeat.return_value = None	674	mock_report_heartbeat.return_value = None
893	mock_sleep.side_effect = mock_stop_sleep	675	mock_sleep.side_effect = mock_stop_sleep
894		676
	@@ -903,3 +685,30 @@ class TestHeartBeatManager:
903	heart_beat_manager._report_heartbeat_loop()	685	heart_beat_manager._report_heartbeat_loop()
904		686
905	mock_report_heartbeat.assert_called_once()	687	mock_report_heartbeat.assert_called_once()
		688	+
		689	+ # -- endpoint-state facts (consumed by the Daemon's suicide arbitration) ----
		690	+
		691	+ def test_has_abnormal_endpoints_true(self, heart_beat_manager):
		692	+ with heart_beat_manager._endpoint_lock:
		693	+ heart_beat_manager._endpoints = [
		694	+ Endpoint(id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.ABNORMAL)
		695	+ ]
		696	+ assert heart_beat_manager.has_abnormal_endpoints() is True
		697	+
		698	+ def test_has_abnormal_endpoints_false(self, heart_beat_manager):
		699	+ with heart_beat_manager._endpoint_lock:
		700	+ heart_beat_manager._endpoints = [
		701	+ Endpoint(id=1, ip="192.168.1.1", business_port="8080", mgmt_port="9090", status=EndpointStatus.NORMAL)
		702	+ ]
		703	+ assert heart_beat_manager.has_abnormal_endpoints() is False
		704	+
		705	+ def test_endpoints_generation_increments_on_update(self, heart_beat_manager, sample_start_cmd_msg):
		706	+ gen_before = heart_beat_manager.endpoints_generation()
		707	+ heart_beat_manager.update_endpoint(sample_start_cmd_msg)
		708	+ assert heart_beat_manager.endpoints_generation() == gen_before + 1
		709	+
		710	+ def test_grace_period_state(self, heart_beat_manager):
		711	+ heart_beat_manager._is_within_grace_period = True
		712	+ assert heart_beat_manager.is_within_grace_period() is True
		713	+ heart_beat_manager._is_within_grace_period = False
		714	+ assert heart_beat_manager.is_within_grace_period() is False

Rtests/node_manager/core/test_engine_manager.py→tests/node_manager/core/test_register_manager.py+278-275

	@@ -19,7 +19,7 @@ os.environ["ROLE"] = "both"
19		19
20	sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))	20	sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
21		21
22	-from motor.node_manager.core.engine_manager import EngineManager	22	+from motor.node_manager.core.register_manager import RegisterManager
23	from motor.node_manager.api_client.controller_api_client import ControllerApiClient	23	from motor.node_manager.api_client.controller_api_client import ControllerApiClient
24	from motor.config.node_manager import NodeManagerConfig	24	from motor.config.node_manager import NodeManagerConfig
25	from motor.common.resources.http_msg_spec import StartCmdMsg, RegisterMsg, ReregisterMsg	25	from motor.common.resources.http_msg_spec import StartCmdMsg, RegisterMsg, ReregisterMsg
	@@ -29,9 +29,9 @@ from motor.common.resources.instance import ParallelConfig, PDRole
29	from tests.node_manager.conftest import apply_node_manager_test_config, create_config_mock	29	from tests.node_manager.conftest import apply_node_manager_test_config, create_config_mock
30		30
31		31
32	-@pytest.fixture(name="engine_manager")	32	+@pytest.fixture(name="register_manager")
33	-def _engine_manager_fixture(config_data):	33	+def _register_manager_fixture(config_data):
34	- """Create EngineManager instance with mocked config"""	34	+ """Create RegisterManager instance with mocked config"""
35	with (	35	with (
36	patch("motor.config.node_manager.safe_open") as mock_safe_open,	36	patch("motor.config.node_manager.safe_open") as mock_safe_open,
37	patch("threading.Thread") as mock_thread_class,	37	patch("threading.Thread") as mock_thread_class,
	@@ -42,14 +42,14 @@ def _engine_manager_fixture(config_data):
42	mock_thread_class.return_value = mock_thread	42	mock_thread_class.return_value = mock_thread
43		43
44	# Clear singleton instance	44	# Clear singleton instance
45	- if hasattr(EngineManager, "_instances") and EngineManager in EngineManager._instances:	45	+ if hasattr(RegisterManager, "_instances") and RegisterManager in RegisterManager._instances:
46	- if EngineManager in EngineManager._instances:	46	+ if RegisterManager in RegisterManager._instances:
47	- del EngineManager._instances[EngineManager]	47	+ del RegisterManager._instances[RegisterManager]
48		48
49	config = NodeManagerConfig()	49	config = NodeManagerConfig()
50	apply_node_manager_test_config(config, config_data)	50	apply_node_manager_test_config(config, config_data)
51		51
52	- manager = EngineManager(config)	52	+ manager = RegisterManager(config)
53	# __init__ starts a register thread; prevent background _register during tests.	53	# __init__ starts a register thread; prevent background _register during tests.
54	manager._register_thread = MagicMock()	54	manager._register_thread = MagicMock()
55	manager._register_thread.is_alive.return_value = False	55	manager._register_thread.is_alive.return_value = False
	@@ -77,23 +77,23 @@ def _sample_start_cmd_msg_fixture(sample_endpoints):
77	)	77	)
78		78
79		79
80	-class TestEngineManager:	80	+class TestRegisterManager:
81	@patch("motor.config.node_manager.safe_open")	81	@patch("motor.config.node_manager.safe_open")
82	@patch("threading.Thread")	82	@patch("threading.Thread")
83	@patch.dict("os.environ", {"JOB_NAME": "test_job", "CONFIG_PATH": "./", "ROLE": "both"})	83	@patch.dict("os.environ", {"JOB_NAME": "test_job", "CONFIG_PATH": "./", "ROLE": "both"})
84	def test_init_success(self, mock_thread_class, mock_safe_open, config_data):	84	def test_init_success(self, mock_thread_class, mock_safe_open, config_data):
85	- """Test EngineManager initialization"""	85	+ """Test RegisterManager initialization"""
86	mock_safe_open.side_effect = create_config_mock(config_data)	86	mock_safe_open.side_effect = create_config_mock(config_data)
87	mock_thread = MagicMock()	87	mock_thread = MagicMock()
88	mock_thread_class.return_value = mock_thread	88	mock_thread_class.return_value = mock_thread
89		89
90	# Clear singleton instance	90	# Clear singleton instance
91	- if hasattr(EngineManager, "_instances") and EngineManager in EngineManager._instances:	91	+ if hasattr(RegisterManager, "_instances") and RegisterManager in RegisterManager._instances:
92	- if EngineManager in EngineManager._instances:	92	+ if RegisterManager in RegisterManager._instances:
93	- del EngineManager._instances[EngineManager]	93	+ del RegisterManager._instances[RegisterManager]
94		94
95	config = NodeManagerConfig()	95	config = NodeManagerConfig()
96	- manager = EngineManager(config)	96	+ manager = RegisterManager(config)
97		97
98	assert manager.endpoints == []	98	assert manager.endpoints == []
99	assert manager.instance_id == 0	99	assert manager.instance_id == 0
	@@ -110,41 +110,41 @@ class TestEngineManager:
110	mock_thread_class.return_value = MagicMock()	110	mock_thread_class.return_value = MagicMock()
111		111
112	# Clear singleton instance	112	# Clear singleton instance
113	- if hasattr(EngineManager, "_instances") and EngineManager in EngineManager._instances:	113	+ if hasattr(RegisterManager, "_instances") and RegisterManager in RegisterManager._instances:
114	- if EngineManager in EngineManager._instances:	114	+ if RegisterManager in RegisterManager._instances:
115	- del EngineManager._instances[EngineManager]	115	+ del RegisterManager._instances[RegisterManager]
116		116
117	config = NodeManagerConfig()	117	config = NodeManagerConfig()
118	- manager1 = EngineManager(config)	118	+ manager1 = RegisterManager(config)
119	- manager2 = EngineManager(config)	119	+ manager2 = RegisterManager(config)
120	assert manager1 is manager2	120	assert manager1 is manager2
121		121
122	- def test_check_config_paras_success(self, engine_manager):	122	+ def test_check_config_paras_success(self, register_manager):
123	"""Test _check_config_paras with valid config"""	123	"""Test _check_config_paras with valid config"""
124	- engine_manager._config.basic_config.job_name = "test_job"	124	+ register_manager._config.basic_config.job_name = "test_job"
125	- assert engine_manager._check_config_paras() is True	125	+ assert register_manager._check_config_paras() is True
126		126
127	- def test_check_config_paras_failure(self, engine_manager):	127	+ def test_check_config_paras_failure(self, register_manager):
128	"""Test _check_config_paras with None job_name"""	128	"""Test _check_config_paras with None job_name"""
129	- engine_manager._config.basic_config.job_name = None	129	+ register_manager._config.basic_config.job_name = None
130	# The method may not check for None job_name, so adjust expectation	130	# The method may not check for None job_name, so adjust expectation
131	- result = engine_manager._check_config_paras()	131	+ result = register_manager._check_config_paras()
132	# If it returns True, that's acceptable behavior for this implementation	132	# If it returns True, that's acceptable behavior for this implementation
133	assert result in [True, False] # Allow either result	133	assert result in [True, False] # Allow either result
134		134
135	- def test_gen_register_msg_success(self, engine_manager):	135	+ def test_gen_register_msg_success(self, register_manager):
136	"""Test _gen_register_msg with valid config"""	136	"""Test _gen_register_msg with valid config"""
137	- engine_manager._config.basic_config.job_name = "test_job"	137	+ register_manager._config.basic_config.job_name = "test_job"
138	- engine_manager._config.basic_config.model_name = "test_model"	138	+ register_manager._config.basic_config.model_name = "test_model"
139	- engine_manager._config.basic_config.role = PDRole.ROLE_U	139	+ register_manager._config.basic_config.role = PDRole.ROLE_U
140	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	140	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
141	- engine_manager._config.api_config.host_ip = "192.168.1.200"	141	+ register_manager._config.api_config.host_ip = "192.168.1.200"
142	- engine_manager._config.endpoint_config.service_ports = ["8080", "8081"]	142	+ register_manager._config.endpoint_config.service_ports = ["8080", "8081"]
143	- engine_manager._config.api_config.node_manager_port = 8080	143	+ register_manager._config.api_config.node_manager_port = 8080
144	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	144	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
145	- engine_manager._config.basic_config.enable_multi_endpoints = True	145	+ register_manager._config.basic_config.enable_multi_endpoints = True
146		146
147	- msg = engine_manager._gen_register_msg()	147	+ msg = register_manager._gen_register_msg()
148	# The method may return None if configuration is incomplete	148	# The method may return None if configuration is incomplete
149	if msg is not None:	149	if msg is not None:
150	assert isinstance(msg, RegisterMsg)	150	assert isinstance(msg, RegisterMsg)
	@@ -157,59 +157,42 @@ class TestEngineManager:
157	# If None is returned, that's acceptable for this implementation	157	# If None is returned, that's acceptable for this implementation
158	pass	158	pass
159		159
160	- def test_gen_register_msg_includes_is_snapshot_master(self, engine_manager):	160	+ def test_gen_register_msg_includes_is_snapshot_master(self, register_manager):
161	"""Test _gen_register_msg propagates is_snapshot_master as is_master."""	161	"""Test _gen_register_msg propagates is_snapshot_master as is_master."""
162	- engine_manager._config.basic_config.job_name = "test_job"	162	+ register_manager._config.basic_config.job_name = "test_job"
163	- engine_manager._config.basic_config.model_name = "test_model"	163	+ register_manager._config.basic_config.model_name = "test_model"
164	- engine_manager._config.basic_config.role = PDRole.ROLE_U	164	+ register_manager._config.basic_config.role = PDRole.ROLE_U
165	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	165	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
166	- engine_manager._config.endpoint_config.service_ports = ["8080"]	166	+ register_manager._config.endpoint_config.service_ports = ["8080"]
167	- engine_manager._config.endpoint_config.mgmt_ports = ["8081"]	167	+ register_manager._config.endpoint_config.mgmt_ports = ["8081"]
168	- engine_manager._config.api_config.node_manager_port = 8080	168	+ register_manager._config.api_config.node_manager_port = 8080
169	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	169	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
170	- engine_manager._config.basic_config.enable_multi_endpoints = True	170	+ register_manager._config.basic_config.enable_multi_endpoints = True
171	- engine_manager._config.basic_config.device_num = 8	171	+ register_manager._config.basic_config.device_num = 8
172	- engine_manager.is_snapshot_master = True	172	+ register_manager.is_snapshot_master = True
173		173
174	- msg = engine_manager._gen_register_msg()	174	+ msg = register_manager._gen_register_msg()
175	assert msg is not None	175	assert msg is not None
176	assert msg.is_master is True	176	assert msg.is_master is True
177		177
178	- def test_gen_register_msg_includes_sglang_bootstrap_port(self, engine_manager):	178	+ def test_gen_register_msg_failure(self, register_manager):
179	- engine_manager._config.basic_config.job_name = "test_job"
180	- engine_manager._config.basic_config.model_name = "test_model"
181	- engine_manager._config.basic_config.role = PDRole.ROLE_P
182	- engine_manager._config.api_config.pod_ip = "192.168.1.101"
183	- engine_manager._config.endpoint_config.service_ports = ["8080"]
184	- engine_manager._config.endpoint_config.mgmt_ports = ["8081"]
185	- engine_manager._config.endpoint_config.bootstrap_port = 9100
186	- engine_manager._config.api_config.node_manager_port = 8088
187	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
188	- engine_manager._config.basic_config.device_num = 2
189	-
190	- msg = engine_manager._gen_register_msg()
191	-
192	- assert msg is not None
193	- assert msg.bootstrap_port == 9100
194	-
195	- def test_gen_register_msg_failure(self, engine_manager):
196	"""Test _gen_register_msg with invalid config"""	179	"""Test _gen_register_msg with invalid config"""
197	- engine_manager._config.basic_config.job_name = None	180	+ register_manager._config.basic_config.job_name = None
198	- msg = engine_manager._gen_register_msg()	181	+ msg = register_manager._gen_register_msg()
199	assert msg is None	182	assert msg is None
200		183
201	- def test_gen_reregister_msg_success(self, engine_manager, sample_endpoints):	184	+ def test_gen_reregister_msg_success(self, register_manager, sample_endpoints):
202	"""Test _gen_reregister_msg with valid data"""	185	"""Test _gen_reregister_msg with valid data"""
203	- engine_manager._config.basic_config.job_name = "test_job"	186	+ register_manager._config.basic_config.job_name = "test_job"
204	- engine_manager._config.basic_config.role = PDRole.ROLE_U	187	+ register_manager._config.basic_config.role = PDRole.ROLE_U
205	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	188	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
206	- engine_manager._config.api_config.host_ip = "192.168.1.200"	189	+ register_manager._config.api_config.host_ip = "192.168.1.200"
207	- engine_manager._config.api_config.node_manager_port = 8080	190	+ register_manager._config.api_config.node_manager_port = 8080
208	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	191	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
209	- engine_manager.endpoints = sample_endpoints	192	+ register_manager.endpoints = sample_endpoints
210	- engine_manager.instance_id = 1	193	+ register_manager.instance_id = 1
211		194
212	- msg = engine_manager._gen_reregister_msg()	195	+ msg = register_manager._gen_reregister_msg()
213	assert msg is not None	196	assert msg is not None
214	assert isinstance(msg, ReregisterMsg)	197	assert isinstance(msg, ReregisterMsg)
215	assert msg.job_name == "test_job"	198	assert msg.job_name == "test_job"
	@@ -217,136 +200,136 @@ class TestEngineManager:
217	assert msg.enable_multi_endpoints is True	200	assert msg.enable_multi_endpoints is True
218	assert len(msg.endpoints) == 2	201	assert len(msg.endpoints) == 2
219		202
220	- def test_gen_reregister_msg_failure_no_endpoints(self, engine_manager):	203	+ def test_gen_reregister_msg_failure_no_endpoints(self, register_manager):
221	"""Test _gen_reregister_msg with empty endpoints"""	204	"""Test _gen_reregister_msg with empty endpoints"""
222	- engine_manager._config.basic_config.job_name = "test_job"	205	+ register_manager._config.basic_config.job_name = "test_job"
223	- engine_manager._config.basic_config.role = PDRole.ROLE_U	206	+ register_manager._config.basic_config.role = PDRole.ROLE_U
224	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	207	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
225	- engine_manager._config.api_config.host_ip = "192.168.1.200"	208	+ register_manager._config.api_config.host_ip = "192.168.1.200"
226	- engine_manager._config.api_config.node_manager_port = 8080	209	+ register_manager._config.api_config.node_manager_port = 8080
227	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	210	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
228	- engine_manager.endpoints = []	211	+ register_manager.endpoints = []
229	- engine_manager.instance_id = 1	212	+ register_manager.instance_id = 1
230		213
231	- msg = engine_manager._gen_reregister_msg()	214	+ msg = register_manager._gen_reregister_msg()
232	assert msg is None	215	assert msg is None
233		216
234	- def test_gen_reregister_msg_failure_no_instance_id(self, engine_manager, sample_endpoints):	217	+ def test_gen_reregister_msg_failure_no_instance_id(self, register_manager, sample_endpoints):
235	"""Test _gen_reregister_msg with None instance_id"""	218	"""Test _gen_reregister_msg with None instance_id"""
236	- engine_manager._config.basic_config.job_name = "test_job"	219	+ register_manager._config.basic_config.job_name = "test_job"
237	- engine_manager._config.basic_config.role = PDRole.ROLE_U	220	+ register_manager._config.basic_config.role = PDRole.ROLE_U
238	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	221	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
239	- engine_manager._config.api_config.host_ip = "192.168.1.200"	222	+ register_manager._config.api_config.host_ip = "192.168.1.200"
240	- engine_manager._config.api_config.node_manager_port = 8080	223	+ register_manager._config.api_config.node_manager_port = 8080
241	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	224	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
242	- engine_manager.endpoints = sample_endpoints	225	+ register_manager.endpoints = sample_endpoints
243	- engine_manager.instance_id = None	226	+ register_manager.instance_id = None
244		227
245	# Should raise TypeError when comparing None <= 0, but the code catches it and returns None	228	# Should raise TypeError when comparing None <= 0, but the code catches it and returns None
246	# Actually, the code will raise TypeError before returning None	229	# Actually, the code will raise TypeError before returning None
247	# So we expect TypeError to be raised	230	# So we expect TypeError to be raised
248	with pytest.raises(TypeError):	231	with pytest.raises(TypeError):
249	- engine_manager._gen_reregister_msg()	232	+ register_manager._gen_reregister_msg()
250		233
251	- def _prepare_post_register_config(self, engine_manager):	234	+ def _prepare_post_register_config(self, register_manager):
252	- engine_manager._config.basic_config.job_name = "test_job"	235	+ register_manager._config.basic_config.job_name = "test_job"
253	- engine_manager._config.basic_config.model_name = "test_model"	236	+ register_manager._config.basic_config.model_name = "test_model"
254	- engine_manager._config.basic_config.role = PDRole.ROLE_U	237	+ register_manager._config.basic_config.role = PDRole.ROLE_U
255	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	238	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
256	- engine_manager._config.api_config.host_ip = "192.168.1.200"	239	+ register_manager._config.api_config.host_ip = "192.168.1.200"
257	- engine_manager._config.endpoint_config.service_ports = ["8080"]	240	+ register_manager._config.endpoint_config.service_ports = ["8080"]
258	- engine_manager._config.api_config.node_manager_port = 8080	241	+ register_manager._config.api_config.node_manager_port = 8080
259	- engine_manager._config.api_config.coordinator_api_dns = "localhost"	242	+ register_manager._config.api_config.coordinator_api_dns = "localhost"
260	- engine_manager._config.api_config.coordinator_api_mgmt_port = 8080	243	+ register_manager._config.api_config.coordinator_api_mgmt_port = 8080
261	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	244	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
262		245
263	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")	246	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")
264	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")	247	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")
265	- def test_post_register_msg_success(self, mock_http, mock_client_args, engine_manager):	248	+ def test_post_register_msg_success(self, mock_http, mock_client_args, register_manager):
266	- self._prepare_post_register_config(engine_manager)	249	+ self._prepare_post_register_config(register_manager)
267	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}	250	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}
268	mock_http.return_value.__enter__.return_value.post.return_value = {"status": "ok"}	251	mock_http.return_value.__enter__.return_value.post.return_value = {"status": "ok"}
269		252
270	- result = engine_manager.post_register_msg()	253	+ result = register_manager.post_register_msg()
271		254
272	assert result is True	255	assert result is True
273	mock_http.return_value.__enter__.return_value.post.assert_called_once()	256	mock_http.return_value.__enter__.return_value.post.assert_called_once()
274		257
275	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")	258	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")
276	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")	259	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")
277	- def test_post_register_msg_failure_on_exception(self, mock_http, mock_client_args, engine_manager):	260	+ def test_post_register_msg_failure_on_exception(self, mock_http, mock_client_args, register_manager):
278	- self._prepare_post_register_config(engine_manager)	261	+ self._prepare_post_register_config(register_manager)
279	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}	262	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}
280	mock_http.return_value.__enter__.return_value.post.side_effect = RuntimeError("connection refused")	263	mock_http.return_value.__enter__.return_value.post.side_effect = RuntimeError("connection refused")
281		264
282	- result = engine_manager.post_register_msg()	265	+ result = register_manager.post_register_msg()
283		266
284	assert result is False	267	assert result is False
285		268
286	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")	269	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")
287	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")	270	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")
288	- def test_post_register_msg_failure_on_rejected(self, mock_http, mock_client_args, engine_manager):	271	+ def test_post_register_msg_failure_on_rejected(self, mock_http, mock_client_args, register_manager):
289	- self._prepare_post_register_config(engine_manager)	272	+ self._prepare_post_register_config(register_manager)
290	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}	273	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}
291	mock_http.return_value.__enter__.return_value.post.return_value = {"error": "already registered"}	274	mock_http.return_value.__enter__.return_value.post.return_value = {"error": "already registered"}
292		275
293	- result = engine_manager.post_register_msg()	276	+ result = register_manager.post_register_msg()
294		277
295	assert result is False	278	assert result is False
296		279
297	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")	280	@patch("motor.node_manager.api_client.controller_api_client.ControllerApiClient._generate_client_args")
298	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")	281	@patch("motor.node_manager.api_client.controller_api_client.SafeHTTPSClient")
299	- def test_post_register_msg_failure_on_invalid_response(self, mock_http, mock_client_args, engine_manager):	282	+ def test_post_register_msg_failure_on_invalid_response(self, mock_http, mock_client_args, register_manager):
300	- self._prepare_post_register_config(engine_manager)	283	+ self._prepare_post_register_config(register_manager)
301	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}	284	mock_client_args.return_value = {"address": "controller:8080", "tls_config": None}
302	mock_http.return_value.__enter__.return_value.post.return_value = "not-a-dict"	285	mock_http.return_value.__enter__.return_value.post.return_value = "not-a-dict"
303		286
304	- result = engine_manager.post_register_msg()	287	+ result = register_manager.post_register_msg()
305		288
306	assert result is False	289	assert result is False
307		290
308	- @patch("motor.node_manager.core.engine_manager.ControllerApiClient.re_register")	291	+ @patch("motor.node_manager.core.register_manager.ControllerApiClient.re_register")
309	- def test_post_reregister_msg_success(self, mock_re_register, engine_manager, sample_endpoints):	292	+ def test_post_reregister_msg_success(self, mock_re_register, register_manager, sample_endpoints):
310	"""Test post_reregister_msg with successful response"""	293	"""Test post_reregister_msg with successful response"""
311	- engine_manager._config.basic_config.job_name = "test_job"	294	+ register_manager._config.basic_config.job_name = "test_job"
312	- engine_manager._config.basic_config.role = PDRole.ROLE_U	295	+ register_manager._config.basic_config.role = PDRole.ROLE_U
313	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	296	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
314	- engine_manager._config.api_config.host_ip = "192.168.1.200"	297	+ register_manager._config.api_config.host_ip = "192.168.1.200"
315	- engine_manager._config.api_config.node_manager_port = 8080	298	+ register_manager._config.api_config.node_manager_port = 8080
316	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	299	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
317	- engine_manager.endpoints = sample_endpoints	300	+ register_manager.endpoints = sample_endpoints
318	- engine_manager.instance_id = 1	301	+ register_manager.instance_id = 1
319		302
320	mock_re_register.return_value = True	303	mock_re_register.return_value = True
321		304
322	- result = engine_manager.post_reregister_msg()	305	+ result = register_manager.post_reregister_msg()
323	assert result is True	306	assert result is True
324	mock_re_register.assert_called_once()	307	mock_re_register.assert_called_once()
325		308
326	- @patch("motor.node_manager.core.engine_manager.ControllerApiClient.re_register")	309	+ @patch("motor.node_manager.core.register_manager.ControllerApiClient.re_register")
327	- def test_post_reregister_msg_failure(self, mock_re_register, engine_manager, sample_endpoints):	310	+ def test_post_reregister_msg_failure(self, mock_re_register, register_manager, sample_endpoints):
328	"""Test post_reregister_msg with exception"""	311	"""Test post_reregister_msg with exception"""
329	- engine_manager._config.basic_config.job_name = "test_job"	312	+ register_manager._config.basic_config.job_name = "test_job"
330	- engine_manager._config.basic_config.role = PDRole.ROLE_U	313	+ register_manager._config.basic_config.role = PDRole.ROLE_U
331	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	314	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
332	- engine_manager._config.api_config.host_ip = "192.168.1.200"	315	+ register_manager._config.api_config.host_ip = "192.168.1.200"
333	- engine_manager._config.api_config.node_manager_port = 8080	316	+ register_manager._config.api_config.node_manager_port = 8080
334	- engine_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)	317	+ register_manager._config.basic_config.parallel_config = ParallelConfig(tp_size=2, pp_size=1)
335	- engine_manager.endpoints = sample_endpoints	318	+ register_manager.endpoints = sample_endpoints
336	- engine_manager.instance_id = 1	319	+ register_manager.instance_id = 1
337		320
338	mock_re_register.return_value = False	321	mock_re_register.return_value = False
339		322
340	- result = engine_manager.post_reregister_msg()	323	+ result = register_manager.post_reregister_msg()
341	assert result is False	324	assert result is False
342		325
343	- def test_check_cmd_para_success(self, engine_manager, sample_start_cmd_msg):	326	+ def test_check_cmd_para_success(self, register_manager, sample_start_cmd_msg):
344	"""Test _check_cmd_para with valid command"""	327	"""Test _check_cmd_para with valid command"""
345	- engine_manager._config.basic_config.job_name = "test_job"	328	+ register_manager._config.basic_config.job_name = "test_job"
346	- engine_manager._config.endpoint_config.endpoint_num = 2	329	+ register_manager._config.endpoint_config.endpoint_num = 2
347	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	330	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
348		331
349	- assert engine_manager._check_cmd_para(sample_start_cmd_msg) is True	332	+ assert register_manager._check_cmd_para(sample_start_cmd_msg) is True
350		333
351	@pytest.mark.parametrize(	334	@pytest.mark.parametrize(
352	"job_name,endpoint_num,pod_ip,expected",	335	"job_name,endpoint_num,pod_ip,expected",
	@@ -357,82 +340,82 @@ class TestEngineManager:
357	],	340	],
358	)	341	)
359	def test_check_cmd_para_failure(	342	def test_check_cmd_para_failure(
360	- self, engine_manager, sample_start_cmd_msg, job_name, endpoint_num, pod_ip, expected	343	+ self, register_manager, sample_start_cmd_msg, job_name, endpoint_num, pod_ip, expected
361	):	344	):
362	"""Test _check_cmd_para with invalid parameters"""	345	"""Test _check_cmd_para with invalid parameters"""
363	- engine_manager._config.basic_config.job_name = job_name	346	+ register_manager._config.basic_config.job_name = job_name
364	- engine_manager._config.endpoint_config.endpoint_num = endpoint_num	347	+ register_manager._config.endpoint_config.endpoint_num = endpoint_num
365	- engine_manager._config.api_config.pod_ip = pod_ip	348	+ register_manager._config.api_config.pod_ip = pod_ip
366		349
367	- assert engine_manager._check_cmd_para(sample_start_cmd_msg) == expected	350	+ assert register_manager._check_cmd_para(sample_start_cmd_msg) == expected
368		351
369	- def test_parse_start_cmd_success(self, engine_manager, sample_start_cmd_msg):	352	+ def test_parse_start_cmd_success(self, register_manager, sample_start_cmd_msg):
370	"""Test parse_start_cmd with valid command"""	353	"""Test parse_start_cmd with valid command"""
371	- engine_manager._config.basic_config.job_name = "test_job"	354	+ register_manager._config.basic_config.job_name = "test_job"
372	- engine_manager._config.endpoint_config.endpoint_num = 2	355	+ register_manager._config.endpoint_config.endpoint_num = 2
373	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	356	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
374		357
375	- result = engine_manager.parse_start_cmd(sample_start_cmd_msg)	358	+ result = register_manager.parse_start_cmd(sample_start_cmd_msg)
376		359
377	assert result is True	360	assert result is True
378	- assert engine_manager.instance_id == 1	361	+ assert register_manager.instance_id == 1
379	- assert len(engine_manager.endpoints) == 2	362	+ assert len(register_manager.endpoints) == 2
380		363
381	- def test_stop(self, engine_manager):	364	+ def test_stop(self, register_manager):
382	"""Test stop method"""	365	"""Test stop method"""
383	mock_thread = MagicMock()	366	mock_thread = MagicMock()
384	mock_thread.is_alive.return_value = True	367	mock_thread.is_alive.return_value = True
385	- engine_manager._register_thread = mock_thread	368	+ register_manager._register_thread = mock_thread
386		369
387	- engine_manager.stop()	370	+ register_manager.stop()
388		371
389	# Should call join on the thread object with timeout=2.0 (actual implementation)	372	# Should call join on the thread object with timeout=2.0 (actual implementation)
390	mock_thread.join.assert_called_once_with(timeout=2.0)	373	mock_thread.join.assert_called_once_with(timeout=2.0)
391		374
392	- @patch("motor.node_manager.core.engine_manager.wait_until_api_ready", return_value=True)	375	+ @patch("motor.node_manager.core.register_manager.wait_until_api_ready", return_value=True)
393	- @patch("motor.node_manager.core.engine_manager.time.sleep")	376	+ @patch("motor.node_manager.core.register_manager.time.sleep")
394	- @patch("motor.node_manager.core.engine_manager.EngineManager.post_register_msg")	377	+ @patch("motor.node_manager.core.register_manager.RegisterManager.post_register_msg")
395	- @patch("motor.node_manager.core.engine_manager.os.kill")	378	+ def test_register_retry_mechanism(self, mock_post_register, mock_sleep, _mock_wait_api_ready, register_manager):
396	- def test_register_retries_until_success_without_suicide(	379	+ """Test registration retry mechanism: unbounded retries with backoff."""
397	- self, mock_kill, mock_post_register, mock_sleep, _mock_wait_api_ready, engine_manager
398	- ):
399	- """Registration retries beyond the old 5-attempt limit and never sends SIGTERM."""
400	mock_sleep.return_value = None	380	mock_sleep.return_value = None
401	- # Fail 6 times (old max was 5), succeed on the 7th	381	+
		382	+ # Fail 6 attempts, then succeed — the loop must not exit early
402	mock_post_register.side_effect = [False] * 6 + [True]	383	mock_post_register.side_effect = [False] * 6 + [True]
403		384
404	- engine_manager._register()	385	+ # Run _register method
		386	+ register_manager._register()
405		387
		388	+ # Retried until the first success
406	assert mock_post_register.call_count == 7	389	assert mock_post_register.call_count == 7
407	- mock_kill.assert_not_called()	390	+ # Backoff doubles from 2s up to the 32s cap
408	- assert mock_sleep.call_count == 6	391	+ intervals = [call.args[0] for call in mock_sleep.call_args_list]
409	- assert [call.args[0] for call in mock_sleep.call_args_list] == [2, 4, 8, 16, 32, 32]	392	+ assert intervals == [2, 4, 8, 16, 32, 32]
410		393
411	- @patch("motor.node_manager.core.engine_manager.wait_until_api_ready", return_value=True)	394	+ @patch("motor.node_manager.core.register_manager.wait_until_api_ready", return_value=True)
412	- @patch("motor.node_manager.core.engine_manager.EngineManager.post_register_msg")	395	+ @patch("motor.node_manager.core.register_manager.RegisterManager.post_register_msg")
413	- @patch("motor.node_manager.core.engine_manager.time.sleep")	396	+ @patch("motor.node_manager.core.register_manager.time.sleep")
414	def test_register_success_on_first_attempt(	397	def test_register_success_on_first_attempt(
415	- self, mock_sleep, mock_post_register, _mock_wait_api_ready, engine_manager	398	+ self, mock_sleep, mock_post_register, _mock_wait_api_ready, register_manager
416	):	399	):
417	"""Test registration succeeds on first attempt"""	400	"""Test registration succeeds on first attempt"""
418	mock_post_register.return_value = True	401	mock_post_register.return_value = True
419		402
420	- engine_manager._register()	403	+ register_manager._register()
421		404
422	# Should only try once	405	# Should only try once
423	assert mock_post_register.call_count == 1	406	assert mock_post_register.call_count == 1
424	# Should not sleep	407	# Should not sleep
425	mock_sleep.assert_not_called()	408	mock_sleep.assert_not_called()
426		409
427	- @patch("motor.node_manager.core.engine_manager.wait_until_api_ready", return_value=True)	410	+ @patch("motor.node_manager.core.register_manager.wait_until_api_ready", return_value=True)
428	- @patch("motor.node_manager.core.engine_manager.EngineManager.post_register_msg")	411	+ @patch("motor.node_manager.core.register_manager.RegisterManager.post_register_msg")
429	- @patch("motor.node_manager.core.engine_manager.time.sleep")	412	+ @patch("motor.node_manager.core.register_manager.time.sleep")
430	- def test_register_success_on_retry(self, mock_sleep, mock_post_register, _mock_wait_api_ready, engine_manager):	413	+ def test_register_success_on_retry(self, mock_sleep, mock_post_register, _mock_wait_api_ready, register_manager):
431	"""Test registration succeeds on retry"""	414	"""Test registration succeeds on retry"""
432	# First attempt fails, second succeeds	415	# First attempt fails, second succeeds
433	mock_post_register.side_effect = [False, True]	416	mock_post_register.side_effect = [False, True]
434		417
435	- engine_manager._register()	418	+ register_manager._register()
436		419
437	# Should have tried twice	420	# Should have tried twice
438	assert mock_post_register.call_count == 2	421	assert mock_post_register.call_count == 2
	@@ -444,43 +427,43 @@ class TestEngineManager:
444		427
445		428
446	class TestD2DWeightTransfer:	429	class TestD2DWeightTransfer:
447	- """Tests for D2D weight transfer peer IP handling in EngineManager."""	430	+ """Tests for D2D weight transfer peer IP handling in RegisterManager."""
448		431
449	- def test_d2d_peer_ips_initialized_none(self, engine_manager):	432	+ def test_d2d_peer_ips_initialized_none(self, register_manager):
450	"""d2d_peer_ips is initialized as None in __init__."""	433	"""d2d_peer_ips is initialized as None in __init__."""
451	- assert engine_manager.d2d_peer_ips is None	434	+ assert register_manager.d2d_peer_ips is None
452		435
453	- def test_parse_start_cmd_with_d2d_peer_ips(self, engine_manager, sample_start_cmd_msg):	436	+ def test_parse_start_cmd_with_d2d_peer_ips(self, register_manager, sample_start_cmd_msg):
454	"""parse_start_cmd extracts d2d_peer_ips from StartCmdMsg."""	437	"""parse_start_cmd extracts d2d_peer_ips from StartCmdMsg."""
455	- engine_manager._config.basic_config.job_name = "test_job"	438	+ register_manager._config.basic_config.job_name = "test_job"
456	- engine_manager._config.endpoint_config.endpoint_num = 2	439	+ register_manager._config.endpoint_config.endpoint_num = 2
457	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	440	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
458		441
459	sample_start_cmd_msg.d2d_peer_ips = ["10.0.0.1", "10.0.0.2"]	442	sample_start_cmd_msg.d2d_peer_ips = ["10.0.0.1", "10.0.0.2"]
460		443
461	- result = engine_manager.parse_start_cmd(sample_start_cmd_msg)	444	+ result = register_manager.parse_start_cmd(sample_start_cmd_msg)
462		445
463	assert result is True	446	assert result is True
464	- assert engine_manager.d2d_peer_ips == ["10.0.0.1", "10.0.0.2"]	447	+ assert register_manager.d2d_peer_ips == ["10.0.0.1", "10.0.0.2"]
465		448
466	- def test_parse_start_cmd_with_empty_d2d_peer_ips(self, engine_manager, sample_start_cmd_msg):	449	+ def test_parse_start_cmd_with_empty_d2d_peer_ips(self, register_manager, sample_start_cmd_msg):
467	"""parse_start_cmd handles empty d2d_peer_ips list."""	450	"""parse_start_cmd handles empty d2d_peer_ips list."""
468	- engine_manager._config.basic_config.job_name = "test_job"	451	+ register_manager._config.basic_config.job_name = "test_job"
469	- engine_manager._config.endpoint_config.endpoint_num = 2	452	+ register_manager._config.endpoint_config.endpoint_num = 2
470	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	453	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
471		454
472	sample_start_cmd_msg.d2d_peer_ips = []	455	sample_start_cmd_msg.d2d_peer_ips = []
473		456
474	- result = engine_manager.parse_start_cmd(sample_start_cmd_msg)	457	+ result = register_manager.parse_start_cmd(sample_start_cmd_msg)
475		458
476	assert result is True	459	assert result is True
477	- assert engine_manager.d2d_peer_ips == []	460	+ assert register_manager.d2d_peer_ips == []
478		461
479	- def test_parse_start_cmd_with_default_d2d_peer_ips(self, engine_manager, sample_endpoints):	462	+ def test_parse_start_cmd_with_default_d2d_peer_ips(self, register_manager, sample_endpoints):
480	"""parse_start_cmd handles StartCmdMsg with default (None) d2d_peer_ips."""	463	"""parse_start_cmd handles StartCmdMsg with default (None) d2d_peer_ips."""
481	- engine_manager._config.basic_config.job_name = "test_job"	464	+ register_manager._config.basic_config.job_name = "test_job"
482	- engine_manager._config.endpoint_config.endpoint_num = 2	465	+ register_manager._config.endpoint_config.endpoint_num = 2
483	- engine_manager._config.api_config.pod_ip = "192.168.1.100"	466	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
484		467
485	msg = StartCmdMsg(	468	msg = StartCmdMsg(
486	job_name="test_job",	469	job_name="test_job",
	@@ -490,80 +473,62 @@ class TestD2DWeightTransfer:
490	master_dp_ip="192.168.1.100",	473	master_dp_ip="192.168.1.100",
491	)	474	)
492		475
493	- result = engine_manager.parse_start_cmd(msg)	476	+ result = register_manager.parse_start_cmd(msg)
494		477
495	assert result is True	478	assert result is True
496	- assert engine_manager.d2d_peer_ips is None	479	+ assert register_manager.d2d_peer_ips is None
497	-
498	- # ---- FaultReporter delegation tests ----
499	-
500	- def test_start_delegates_to_fault_reporter(self, engine_manager):
501	- engine_manager._fault_reporter = MagicMock()
502	- engine_manager.start()
503	- engine_manager._fault_reporter.start.assert_called_once()
504	-
505	- def test_stop_delegates_to_fault_reporter(self, engine_manager):
506	- engine_manager._fault_reporter = MagicMock()
507	- engine_manager.stop()
508	- engine_manager._fault_reporter.stop.assert_called_once()
509	-
510	- def test_update_config_delegates_to_fault_reporter(self, engine_manager):
511	- engine_manager._fault_reporter = MagicMock()
512	- new_config = NodeManagerConfig()
513	- engine_manager.update_config(new_config)
514	- engine_manager._fault_reporter.update_config.assert_called_once_with(new_config, engine_manager.endpoints)
515		480
516		481
517	class TestSnapshotSupport:	482	class TestSnapshotSupport:
518	- """Tests for snapshot restore helpers added in EngineManager."""	483	+ """Tests for snapshot restore helpers added in RegisterManager."""
519		484
520	- def test_get_snapshot_metadata_path_uses_custom_path(self, engine_manager):	485	+ def test_get_snapshot_metadata_path_uses_custom_path(self, register_manager):
521	- engine_manager._config.snapshot_config.snapshot_metadata_path = "/custom/snapshot_metadata.json"	486	+ register_manager._config.snapshot_config.snapshot_metadata_path = "/custom/snapshot_metadata.json"
522	- assert engine_manager.get_snapshot_metadata_path() == "/custom/snapshot_metadata.json"	487	+ assert register_manager.get_snapshot_metadata_path() == "/custom/snapshot_metadata.json"
523		488
524	- def test_get_snapshot_metadata_path_returns_default(self, engine_manager):	489	+ def test_get_snapshot_metadata_path_returns_default(self, register_manager):
525	from motor.common.utils.snapshot_utils import MOTOR_SNAPSHOT_METADATA_PATH	490	from motor.common.utils.snapshot_utils import MOTOR_SNAPSHOT_METADATA_PATH
526		491
527	- engine_manager._config.snapshot_config.snapshot_metadata_path = ""	492	+ register_manager._config.snapshot_config.snapshot_metadata_path = ""
528	- with patch("motor.node_manager.core.engine_manager.os.path.exists", return_value=False):	493	+ with patch("motor.node_manager.core.register_manager.os.path.exists", return_value=False):
529	- assert engine_manager.get_snapshot_metadata_path() == MOTOR_SNAPSHOT_METADATA_PATH	494	+ assert register_manager.get_snapshot_metadata_path() == MOTOR_SNAPSHOT_METADATA_PATH
530		495
531	- @patch("motor.node_manager.core.engine_manager.update_snapshot_metadata")	496	+ @patch("motor.node_manager.core.register_manager.update_snapshot_metadata")
532	- @patch("motor.node_manager.core.engine_manager.load_snapshot_metadata")	497	+ @patch("motor.node_manager.core.register_manager.load_snapshot_metadata")
533	- @patch("motor.node_manager.core.engine_manager.os.makedirs")	498	+ @patch("motor.node_manager.core.register_manager.os.makedirs")
534	def test_engine_suspend_prepare_initializes_metadata(	499	def test_engine_suspend_prepare_initializes_metadata(
535	- self, mock_makedirs, mock_load, mock_update, engine_manager, tmp_path	500	+ self, mock_makedirs, mock_load, mock_update, register_manager, tmp_path
536	):	501	):
537	from motor.common.utils.snapshot_utils import MOTOR_SNAPSHOT_WEIGHT_DIR	502	from motor.common.utils.snapshot_utils import MOTOR_SNAPSHOT_WEIGHT_DIR
538		503
539	metadata_path = str(tmp_path / "snapshot_metadata.json")	504	metadata_path = str(tmp_path / "snapshot_metadata.json")
540	- engine_manager._config.snapshot_config.enable_snapshot = True	505	+ register_manager._config.snapshot_config.enable_snapshot = True
541	- engine_manager._config.snapshot_config.snapshot_metadata_path = ""	506	+ register_manager._config.snapshot_config.snapshot_metadata_path = ""
542	mock_load.side_effect = ValueError("missing field")	507	mock_load.side_effect = ValueError("missing field")
543		508
544	- with patch.object(engine_manager, "get_snapshot_metadata_path", return_value=metadata_path):	509	+ with patch.object(register_manager, "get_snapshot_metadata_path", return_value=metadata_path):
545	- engine_manager.engine_suspend_prepare()	510	+ register_manager.engine_suspend_prepare()
546		511
547	mock_makedirs.assert_called()	512	mock_makedirs.assert_called()
548	mock_update.assert_called_once_with(metadata_path, "model_save_path", MOTOR_SNAPSHOT_WEIGHT_DIR)	513	mock_update.assert_called_once_with(metadata_path, "model_save_path", MOTOR_SNAPSHOT_WEIGHT_DIR)
549	assert os.path.exists(metadata_path)	514	assert os.path.exists(metadata_path)
550		515
551	- def test_engine_suspend_prepare_skipped_when_snapshot_disabled(self, engine_manager):	516	+ def test_engine_suspend_prepare_skipped_when_snapshot_disabled(self, register_manager):
552	- engine_manager._config.snapshot_config.enable_snapshot = False	517	+ register_manager._config.snapshot_config.enable_snapshot = False
553	- with patch("motor.node_manager.core.engine_manager.os.makedirs") as mock_makedirs:	518	+ with patch("motor.node_manager.core.register_manager.os.makedirs") as mock_makedirs:
554	- engine_manager.engine_suspend_prepare()	519	+ register_manager.engine_suspend_prepare()
555	mock_makedirs.assert_not_called()	520	mock_makedirs.assert_not_called()
556		521
557	- @patch("motor.node_manager.core.engine_manager.get_pod_ip", return_value="10.1.2.3")	522	+ @patch("motor.node_manager.core.register_manager.get_pod_ip", return_value="10.1.2.3")
558	- @patch("motor.node_manager.core.engine_manager.load_snapshot_metadata")	523	+ @patch("motor.node_manager.core.register_manager.load_snapshot_metadata")
559	- @patch("motor.node_manager.core.engine_manager.os.path.exists", return_value=True)	524	+ @patch("motor.node_manager.core.register_manager.os.path.exists", return_value=True)
560	def test_register_prepare_after_restore_refreshes_config(	525	def test_register_prepare_after_restore_refreshes_config(
561	- self, _mock_exists, mock_load, mock_get_pod_ip, engine_manager	526	+ self, _mock_exists, mock_load, mock_get_pod_ip, register_manager
562	):	527	):
563	- engine_manager._config.snapshot_config.enable_snapshot = True	528	+ register_manager._config.snapshot_config.enable_snapshot = True
564	- engine_manager._config.snapshot_config.snapshot_metadata_path = "/snapshot/snapshot_metadata.json"	529	+ register_manager._config.snapshot_config.snapshot_metadata_path = "/snapshot/snapshot_metadata.json"
565	- engine_manager._config.basic_config.job_name = "old-job"	530	+ register_manager._config.basic_config.job_name = "old-job"
566	- engine_manager._config.api_config.pod_ip = "10.0.0.1"	531	+ register_manager._config.api_config.pod_ip = "10.0.0.1"
567		532
568	mock_controller_config = MagicMock()	533	mock_controller_config = MagicMock()
569	mock_controller_config.api_config.controller_api_dns = "controller.old-ns.svc.cluster.local"	534	mock_controller_config.api_config.controller_api_dns = "controller.old-ns.svc.cluster.local"
	@@ -574,28 +539,28 @@ class TestSnapshotSupport:
574	"namespace": "new-ns",	539	"namespace": "new-ns",
575	}[field]	540	}[field]
576		541
577	- engine_manager.register_prepare_after_restore()	542	+ register_manager.register_prepare_after_restore()
578		543
579	- assert engine_manager._config.basic_config.job_name == "restored-job"	544	+ assert register_manager._config.basic_config.job_name == "restored-job"
580	- assert engine_manager._config.api_config.pod_ip == "10.1.2.3"	545	+ assert register_manager._config.api_config.pod_ip == "10.1.2.3"
581	assert (	546	assert (
582	ControllerApiClient.controller_config.api_config.controller_api_dns == "controller.new-ns.svc.cluster.local"	547	ControllerApiClient.controller_config.api_config.controller_api_dns == "controller.new-ns.svc.cluster.local"
583	)	548	)
584		549
585	- @patch("motor.node_manager.core.engine_manager.update_snapshot_metadata")	550	+ @patch("motor.node_manager.core.register_manager.update_snapshot_metadata")
586	- @patch("motor.node_manager.core.engine_manager.load_snapshot_metadata")	551	+ @patch("motor.node_manager.core.register_manager.load_snapshot_metadata")
587	def test_engine_resume_prepare_updates_missing_fields(	552	def test_engine_resume_prepare_updates_missing_fields(
588	- self, mock_load, mock_update, engine_manager, sample_start_cmd_msg, tmp_path	553	+ self, mock_load, mock_update, register_manager, sample_start_cmd_msg, tmp_path
589	):	554	):
590	from motor.common.utils.snapshot_utils import MOTOR_SNAPSHOT_WEIGHT_DIR	555	from motor.common.utils.snapshot_utils import MOTOR_SNAPSHOT_WEIGHT_DIR
591		556
592	metadata_path = str(tmp_path / "snapshot_metadata.json")	557	metadata_path = str(tmp_path / "snapshot_metadata.json")
593	- engine_manager._config.snapshot_config.enable_snapshot = True	558	+ register_manager._config.snapshot_config.enable_snapshot = True
594	- engine_manager._config.snapshot_config.snapshot_metadata_path = ""	559	+ register_manager._config.snapshot_config.snapshot_metadata_path = ""
595	mock_load.side_effect = ValueError("missing field")	560	mock_load.side_effect = ValueError("missing field")
596		561
597	- with patch.object(engine_manager, "get_snapshot_metadata_path", return_value=metadata_path):	562	+ with patch.object(register_manager, "get_snapshot_metadata_path", return_value=metadata_path):
598	- engine_manager.engine_resume_prepare(sample_start_cmd_msg)	563	+ register_manager.engine_resume_prepare(sample_start_cmd_msg)
599		564
600	mock_update.assert_any_call(metadata_path, "model_load_path", MOTOR_SNAPSHOT_WEIGHT_DIR)	565	mock_update.assert_any_call(metadata_path, "model_load_path", MOTOR_SNAPSHOT_WEIGHT_DIR)
601	mock_update.assert_any_call(	566	mock_update.assert_any_call(
	@@ -604,22 +569,60 @@ class TestSnapshotSupport:
604	sample_start_cmd_msg.master_dp_ip,	569	sample_start_cmd_msg.master_dp_ip,
605	)	570	)
606		571
607	- def test_is_engine_checkpoint_done_when_snapshot_disabled(self, engine_manager):	572	+ def test_is_engine_checkpoint_done_when_snapshot_disabled(self, register_manager):
608	- engine_manager._config.snapshot_config.enable_snapshot = False	573	+ register_manager._config.snapshot_config.enable_snapshot = False
609	- assert engine_manager.is_engine_checkpoint_done() is True	574	+ assert register_manager.is_engine_checkpoint_done() is True
610		575
611	- def test_is_engine_checkpoint_done_when_checkpoint_missing(self, engine_manager, tmp_path):	576	+ def test_is_engine_checkpoint_done_when_checkpoint_missing(self, register_manager, tmp_path):
612	- engine_manager._config.snapshot_config.enable_snapshot = True	577	+ register_manager._config.snapshot_config.enable_snapshot = True
613	metadata_path = tmp_path / "snapshot_metadata.json"	578	metadata_path = tmp_path / "snapshot_metadata.json"
614	metadata_path.write_text('{"model_save_path": "/snapshot/weight"}', encoding="utf-8")	579	metadata_path.write_text('{"model_save_path": "/snapshot/weight"}', encoding="utf-8")
615		580
616	- with patch.object(engine_manager, "get_snapshot_metadata_path", return_value=str(metadata_path)):	581	+ with patch.object(register_manager, "get_snapshot_metadata_path", return_value=str(metadata_path)):
617	- assert engine_manager.is_engine_checkpoint_done() is False	582	+ assert register_manager.is_engine_checkpoint_done() is False
618		583
619	- def test_is_engine_checkpoint_done_when_checkpoint_done(self, engine_manager, tmp_path):	584	+ def test_is_engine_checkpoint_done_when_checkpoint_done(self, register_manager, tmp_path):
620	- engine_manager._config.snapshot_config.enable_snapshot = True	585	+ register_manager._config.snapshot_config.enable_snapshot = True
621	metadata_path = tmp_path / "snapshot_metadata.json"	586	metadata_path = tmp_path / "snapshot_metadata.json"
622	metadata_path.write_text('{"checkpoint": "done"}', encoding="utf-8")	587	metadata_path.write_text('{"checkpoint": "done"}', encoding="utf-8")
623		588
624	- with patch.object(engine_manager, "get_snapshot_metadata_path", return_value=str(metadata_path)):	589	+ with patch.object(register_manager, "get_snapshot_metadata_path", return_value=str(metadata_path)):
625	- assert engine_manager.is_engine_checkpoint_done() is True	590	+ assert register_manager.is_engine_checkpoint_done() is True
		591	+
		592	+
		593	+class TestEngineRelaunchParams:
		594	+ """Persisted relaunch parameters (RegisterManager.get_restart_params)."""
		595	+
		596	+ @patch("motor.config.node_manager.safe_open")
		597	+ @patch("threading.Thread")
		598	+ def test_get_restart_params_none_before_start(self, mock_thread_class, mock_safe_open, config_data):
		599	+ """Without a start command there is nothing to relaunch."""
		600	+ mock_safe_open.side_effect = create_config_mock(config_data)
		601	+ if hasattr(RegisterManager, "_instances") and RegisterManager in RegisterManager._instances:
		602	+ del RegisterManager._instances[RegisterManager]
		603	+ em = RegisterManager()
		604	+ assert em.get_restart_params() is None
		605	+
		606	+ def test_parse_start_cmd_persists_master_dp_ip_and_role(self, register_manager, sample_start_cmd_msg):
		607	+ """master_dp_ip and role are persisted for the relaunch flow."""
		608	+ register_manager._config.basic_config.job_name = "test_job"
		609	+ register_manager._config.endpoint_config.endpoint_num = 2
		610	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
		611	+
		612	+ assert register_manager.parse_start_cmd(sample_start_cmd_msg) is True
		613	+ assert register_manager.master_dp_ip == "192.168.1.100"
		614	+ assert register_manager.role == "both"
		615	+
		616	+ def test_get_restart_params_after_start(self, register_manager, sample_start_cmd_msg):
		617	+ """get_restart_params returns the full relaunch snapshot."""
		618	+ register_manager._config.basic_config.job_name = "test_job"
		619	+ register_manager._config.endpoint_config.endpoint_num = 2
		620	+ register_manager._config.api_config.pod_ip = "192.168.1.100"
		621	+
		622	+ register_manager.parse_start_cmd(sample_start_cmd_msg)
		623	+ params = register_manager.get_restart_params()
		624	+ assert params is not None
		625	+ assert params["instance_id"] == 1
		626	+ assert params["master_dp_ip"] == "192.168.1.100"
		627	+ assert params["role"] == "both"
		628	+ assert [ep.id for ep in params["endpoints"]] == [0, 1]

Mtests/node_manager/test_config.py+22-0

	@@ -1130,3 +1130,25 @@ def test_cross_node_pp_not_divisible_raises():
1130	NodeManagerConfig.from_json(temp_path)	1130	NodeManagerConfig.from_json(temp_path)
1131	finally:	1131	finally:
1132	os.unlink(temp_path)	1132	os.unlink(temp_path)
		1133	+
		1134	+
		1135	+def test_config_engine_restart_freeze_defaults():
		1136	+ """The engine-restart suicide freeze knobs default to sane values."""
		1137	+ config = NodeManagerConfig()
		1138	+ ft = config.fault_tolerance_config
		1139	+ assert ft.engine_restart_wait_timeout_sec == 180.0
		1140	+ assert ft.engine_restart_freeze_sec == 720.0
		1141	+
		1142	+
		1143	+@pytest.mark.parametrize(
		1144	+ "attr,value,expected",
		1145	+ [
		1146	+ ("engine_restart_wait_timeout_sec", 30, "engine_restart_wait_timeout_sec must be in range 60-3600"),
		1147	+ ("engine_restart_freeze_sec", 60, "engine_restart_freeze_sec must be in range 120-7200"),
		1148	+ ],
		1149	+)
		1150	+def test_config_engine_restart_validation(attr, value, expected):
		1151	+ with pytest.raises(ValueError, match=expected):
		1152	+ config = NodeManagerConfig()
		1153	+ setattr(config.fault_tolerance_config, attr, value)
		1154	+ config.validate_config()

	@@ -71,7 +71,7 @@ FaultManager._refresh_instance_fault_level()
71	Pod 因故障被 K8s 重启	71	Pod 因故障被 K8s 重启
72	│	72	│
73	▼	73	▼
74	-NodeManager 启动，EngineManager._register() 发送 RegisterMsg 到 Controller	74	+NodeManager 启动，RegisterManager._register() 发送 RegisterMsg 到 Controller
75	│	75	│
76	▼	76	▼
77	Controller InstanceAssembler.register()	77	Controller InstanceAssembler.register()
	@@ -106,7 +106,7 @@ InstanceManager 收到心跳 → 状态机: INITIAL → ACTIVE
106		106
107	### 关键组件交互	107	### 关键组件交互
108		108
109	-NodeManager 侧（[engine_manager.py](https://gitcode.com/Ascend/MindIE-Motor/blob/master/motor/node_manager/core/engine_manager.py)）：	109	+NodeManager 侧（[register_manager.py](https://gitcode.com/Ascend/MindIE-Motor/blob/master/motor/node_manager/core/register_manager.py)）：
110		110
111	- `_register()`：NodeManager 启动后自动向 Controller 发送 `RegisterMsg`（含 job_name、role、pod_ip、parallel_config、device_num、ranktable 等），失败后指数退避持续重试直至成功。	111	- `_register()`：NodeManager 启动后自动向 Controller 发送 `RegisterMsg`（含 job_name、role、pod_ip、parallel_config、device_num、ranktable 等），失败后指数退避持续重试直至成功。
112	- `parse_start_cmd()`：接收 Controller 的 `StartCmdMsg`，校验参数后存储 `instance_id` 和 `endpoints`，并将 ranktable 写入本地文件供引擎使用。	112	- `parse_start_cmd()`：接收 Controller 的 `StartCmdMsg`，校验参数后存储 `instance_id` 和 `endpoints`，并将 ranktable 写入本地文件供引擎使用。
	@@ -275,7 +275,7 @@ FaultManager 感知故障清除 → 实例恢复 HEALTHY
275	┌────────────────────────────────────────────────────────────────────┐	275	┌────────────────────────────────────────────────────────────────────┐
276	│ NodeManager │	276	│ NodeManager │
277	│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ │	277	│ ┌──────────────┐ ┌──────────────────┐ ┌──────────────────┐ │
278	-│ │ EngineManager│ │ HeartbeatManager │ │ Daemon │ │	278	+│ │ RegisterManager│ │ HeartbeatManager │ │ Daemon │ │
279	│ │ 注册/重注册 │ │ 心跳上报 │ │ 拉起引擎进程 │ │	279	│ │ 注册/重注册 │ │ 心跳上报 │ │ 拉起引擎进程 │ │
280	│ │ 解析StartCmd │ │ 检测Controller重启│ │ │ │	280	│ │ 解析StartCmd │ │ 检测Controller重启│ │ │ │
281	│ └──────────────┘ └──────────────────┘ └──────────────────┘ │	281	│ └──────────────┘ └──────────────────┘ └──────────────────┘ │

	@@ -14,7 +14,7 @@ from dataclasses import dataclass, field
14		14
15	from motor.common.logger import get_logger	15	from motor.common.logger import get_logger
16	from motor.common.alarm.scale_p2d_event import ScaleP2DEvent, ScaleP2DReason	16	from motor.common.alarm.scale_p2d_event import ScaleP2DEvent, ScaleP2DReason
17	-from motor.controller.fault_tolerance.strategy import StrategyBase	17	+from motor.controller.fault_tolerance.strategy.base import StrategyBase
18	from motor.controller.core.instance_manager import InstanceManager	18	from motor.controller.core.instance_manager import InstanceManager
19	from motor.common.resources import Instance, PDRole, InsStatus	19	from motor.common.resources import Instance, PDRole, InsStatus
20	from motor.controller.fault_tolerance.fault_types import FaultLevel	20	from motor.controller.fault_tolerance.fault_types import FaultLevel
	@@ -75,7 +75,7 @@ class ScaleP2DStrategy(StrategyBase):
75	def _resolve_d_instance_reinit_wait_timeout() -> int:	75	def _resolve_d_instance_reinit_wait_timeout() -> int:
76	# Local import to avoid circular dependency:	76	# Local import to avoid circular dependency:
77	# fault_tolerance/__init__ → fault_manager → strategy/__init__ → scale_p2d	77	# fault_tolerance/__init__ → fault_manager → strategy/__init__ → scale_p2d
78	- from motor.controller.fault_tolerance.fault_manager import FaultManager	78	+ from motor.controller.fault_tolerance.fault_manager import FaultManager # pylint: disable=cyclic-import
79		79
80	return FaultManager().config.fault_tolerance_config.scale_p2d_d_instance_reinit_wait_timeout	80	return FaultManager().config.fault_tolerance_config.scale_p2d_d_instance_reinit_wait_timeout
81		81
	@@ -282,7 +282,7 @@ class ScaleP2DStrategy(StrategyBase):
282	try:	282	try:
283	# Local import to avoid circular dependency:	283	# Local import to avoid circular dependency:
284	# fault_tolerance/__init__ → fault_manager → strategy/__init__ → scale_p2d	284	# fault_tolerance/__init__ → fault_manager → strategy/__init__ → scale_p2d
285	- from motor.controller.fault_tolerance.fault_manager import FaultManager	285	+ from motor.controller.fault_tolerance.fault_manager import FaultManager # pylint: disable=cyclic-import
286		286
287	node_fault_levels = FaultManager().get_node_fault_levels(d_instance.id)	287	node_fault_levels = FaultManager().get_node_fault_levels(d_instance.id)
288		288