Star3971
890
代码
代码
Issues209
Pull Requests250
项目讨论
Wiki
分析
Star3971
890
已合并
feat(online-rl): 打通结合AIGW任务采集、策略训练与跨进程验证闭环 #2543
mengyuanli创建于 9 天前
feat(online-rl): 打通结合AIGW任务采集、策略训练与跨进程验证闭环 #2543
已合并
mengyuanli创建于 9 天前
从mengyuanli/agent-coreagentos-v1_3合入到openJiuwen/agent-coredevelop
共 130 个文件变更+9708-13439
Adocs/dev/online-rl-service-operations.md+133-0
@@ -0,0 +1,133 @@
1+# Online RL Service Operations
2+ 
3+AgentBox Adapter (AIGW) is the only production inference gateway. It owns the loopback RL Service process and exposes
4+the public Service, Task, Training Run, trajectory, and LoRA control APIs. The RL Service does not start an Agent, vLLM,
5+Judge, Redis, or a second production gateway.
6+ 
7+## Prerequisites
8+ 
9+- AIGW and agent-core must run on the same host.
10+- Redis must use a dedicated database or dedicated instance. Do not use `FLUSHDB`; delete only owned keys when doing
11+ manual maintenance.
12+- AIGW and the RL Service must see the same absolute `lora.root`. The PPO worker writes version directories below
13+ that root and vLLM reads those exact paths.
14+- The RL Service endpoint must listen on loopback. HMAC protects the public AIGW control routes; internal loopback
15+ callbacks are not public routes.
16+ 
17+## RL Service Config
18+ 
19+Save a production config such as `/etc/openjiuwen/rl-service.yaml`:
20+ 
21+```yaml
22+listen_host: 127.0.0.1
23+listen_port: 18081
24+redis_url: redis://127.0.0.1:6379/8
25+trajectory_retention_seconds: 604800
26+model_id: Qwen3-8B
27+base_model_path: /models/Qwen3-8B
28+aigw_endpoint: http://127.0.0.1:18080
29+judge_endpoint: http://127.0.0.1:18082/v1/chat/completions
30+judge_model: Qwen3-8B-Judge
31+judge_api_key: EMPTY
32+judge_votes: 3
33+judge_retries: 2
34+judge_timeout: 30
35+lora_activation_timeout: 150
36+min_samples_for_training: 32
37+max_samples_per_run: 128
38+ppo_samples_per_step: 32
39+ppo_config_path: /etc/openjiuwen/ppo.yaml
40+nproc_per_node: 4
41+training_gpu_ids: "4,5,6,7"
42+lora_repository_path: /srv/online-rl/loras
43+record_dir: /var/lib/openjiuwen/online-rl/records
44+log_path: /var/log/openjiuwen/rl-service.log
45+log_max_bytes: 10485760
46+log_backup_count: 5
47+log_level: INFO
48+```
49+ 
50+Set `lora_activation_timeout` higher than AIGW's `lora.operationTimeout` so AIGW can finish a multi-instance load or
51+rollback before the RL Service decides the Training Run result.
52+ 
53+The AIGW config for the same model must include:
54+ 
55+```json
56+{
57+ "lora": {
58+ "root": "/srv/online-rl/loras",
59+ "statePath": "/var/lib/aigw/lora-state.json",
60+ "operationTimeout": "60s"
61+ },
62+ "onlineRL": {
63+ "command": ["/opt/openjiuwen/bin/python", "-m", "openjiuwen.agent_evolving.agent_rl.online.service"],
64+ "configPath": "/etc/openjiuwen/rl-service.yaml",
65+ "endpoint": "http://127.0.0.1:18081",
66+ "hookTimeout": "30s",
67+ "hookRetries": 2,
68+ "drainTimeout": "120s",
69+ "controlTimeout": "300s"
70+ }
71+}
72+```
73+ 
74+Configure `global.cryptoSock` and the existing AIGW key provider in production. Control signatures use
75+`hex(HMAC-SHA256(apiHmacKey, X-Timestamp + raw_request_body))`, where `X-Timestamp` is Unix milliseconds.
76+ 
77+## Start And Stop
78+ 
79+Start AIGW normally; do not start the RL Service separately:
80+ 
81+```bash
82+/usr/local/bin/aigw --config=/etc/aigw/conf/aigw.json
83+```
84+ 
85+Call signed `POST /v1/rl/service/start`, then check `GET /v1/rl/service`. A ready response has `status=running` and
86+means both `/health` and Redis were ready. Call signed `POST /v1/rl/service/stop` before stopping AIGW. Stop drains
87+Task captures, cancels an active Training Run, and terminates only the owned child process. AIGW also performs this
88+cleanup on normal shutdown.
89+ 
90+RL Service logs are written to the configured `log_path` (`/var/log/openjiuwen/rl-service.log` above). AIGW logs
91+process start, readiness, abnormal exit, hook retries, drain failures, and LoRA operations in its normal log directory.
92+After an abnormal RL Service exit, `GET /v1/rl/service` reports `failed`; ordinary inference and LoRA control remain
93+available. Correct the cause and call Service start again. A Run interrupted in `queued` or `training` is marked failed
94+with `service_restarted` and its fixed samples return to pending; an `activating` Run retries idempotent activation.
95+ 
96+## JiuwenSwarm Online Update Example
97+ 
98+For a runnable Task, terminal-reward, Training Run, and policy-verification flow, see
99+[`examples/jiuwenrl_online`](../../examples/jiuwenrl_online/README.md). The example configures JiuwenSwarm inference to
100+use AIGW and keeps service deployment separate from per-interaction RL control.
101+ 
102+## No-GPU System Verification
103+ 
104+From agent-core, run:
105+ 
106+```bash
107+bash tests/system_tests/agent_evolving/agent_rl/online/run_aigw_system.sh
108+```
109+ 
110+The command uses a real AIGW binary from AgentBox Adapter, a real RL Service process, and a dedicated temporary Redis
111+Docker container. Only vLLM, Judge, and PPO are fake. Set `AIGW_REPO`, `AIGW_BIN`, or an explicitly isolated
112+`ONLINE_RL_REDIS_URL` to override defaults. The harness never runs `FLUSHDB`; it removes only the container it created
113+and explicitly terminates every process/server it owns. The Redis pause/recovery case is skipped when an external
114+Redis URL is supplied because the harness does not own that server. No GPU or real PPO installation is required.
115+ 
116+## Manual PPO To LoRA Acceptance
117+ 
118+1. Start Redis, real inference vLLM instances, the Judge, and AIGW. Register every vLLM instance for the configured
119+ `model_id`, then start the RL Service through AIGW.
120+2. Start terminal and delayed-feedback Tasks through AIGW. Send Agent inference with stable
121+ `X-Agent-Session-Id` and, for delayed feedback, `X-Agent-Turn-Id`. Stop the Tasks and submit terminal rewards.
122+3. Confirm trajectory stats and details contain prompt/completion token IDs, aligned log probabilities, usage,
123+ finish reason, tool calls when present, reward, and the captured policy version.
124+4. Explicitly `POST /v1/rl/training/runs`. Confirm `sample_count` and `policy_versions` are the fixed batch present at
125+ creation; samples arriving later must remain pending for the next Run.
126+5. Observe `queued -> training -> activating -> succeeded`. Verify the artifact is a new
127+ `/srv/online-rl/loras/<model>/vN` directory and all registered vLLM instances accepted `load_lora_adapter`.
128+6. Start a new Task and confirm inference uses `<model>:vN`. A Task started before activation must remain pinned to its
129+ old policy. Verify LoRA requests bypass prefix-cache routing.
130+7. Delete the active LoRA. Confirm new requests return to base immediately, pinned Tasks finish on the old version,
131+ and `unload_lora_adapter` occurs only after their leases drain.
132+8. Restart AIGW and confirm LoRA state recovers as `active` or explicitly `degraded`; never silently route a missing
133+ artifact. Stop the RL Service and confirm ordinary OpenAI/Anthropic inference and LoRA control still work.
Mdocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/agent_rl.README.md+7-9
@@ -45,18 +45,16 @@
45 45 
46| CLASS / FUNCTION | DESCRIPTION |46| CLASS / FUNCTION | DESCRIPTION |
47|------------------|-------------|47|------------------|-------------|
48-| [GatewayConfig](./online/gateway.md) | Online-RL Gateway runtime config dataclass. |48+| `RLServiceConfig` | Static loopback RL Service configuration. |
49-| [build_app_from_config](./online/gateway.md) | Production entry that assembles a FastAPI app from a `GatewayConfig`. |49+| `build_rl_service_app` | RL Service HTTP assembly for Tasks, captures, trajectories, and Training Runs. |
50-| [build_gateway_app](./online/gateway.md) | FastAPI assembly registering routes (`/health`, `/v1/gateway/stats`, `/v1/gateway/upload/batch`, `/v1/chat/completions`, catch-all proxy). |50+| `TaskRegistry` | Redis-backed owner of Task, turn, capture, and reward state. |
51-| [InferenceNotifier](./online/inference.md) | vLLM LoRA hot-loading notifier. |51+| `CapturePipeline` | Complete OpenAI request/response validation and rewarded trajectory publication. |
52| [JudgeScorer](./online/judge.md) | High-level async LLM-as-a-Judge scoring client. |52| [JudgeScorer](./online/judge.md) | High-level async LLM-as-a-Judge scoring client. |
53-| [evaluate_judge_scores](./online/judge.md) | Core scoring entry; multi-vote averaging normalized to `[-1, 1]`. |53+| [evaluate_judge_scores](./online/judge.md) | Core scoring entry; multi-vote averaging normalized to `[0, 1]`. |
54-| [LauncherPaths](./online/launcher.md) | Path-layout dataclass for the online RL loop orchestration. |
55-| [run_online_rl_loop](./online/launcher.md) | Top-level orchestration entry that spawns and supervises all service processes. |
56| [RLOnlineRail](./online/rail.md) | Online RL trajectory collection rail hooking into the agent lifecycle. |54| [RLOnlineRail](./online/rail.md) | Online RL trajectory collection rail hooking into the agent lifecycle. |
57| [TrajectoryUploader](./online/rail.md) | Async uploader of rail-v1 batches to the gateway. |55| [TrajectoryUploader](./online/rail.md) | Async uploader of rail-v1 batches to the gateway. |
58-| [OnlineTrainingScheduler](./online/scheduler.md) | Background-thread scheduler polling Redis and triggering PPO training batches. |56+| `TrainingRunner` | Explicit fixed-batch PPO, LoRA activation, cancellation, and recovery lifecycle. |
59-| [PPOTrainingExecutor](./online/scheduler.md) | Ray/verl PPO runner lifecycle and batch executor. |57+| `PPOTrainingExecutor` | Ray/verl PPO adapter used by an explicit Training Run. |
60 58 
61**Functions**:59**Functions**:
62 60 
Ddocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/online/gateway.md+0-483
@@ -1,483 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.gateway
2- 
3-Online-RL Gateway: a FastAPI reverse-proxy that sits in front of an LLM inference endpoint, records per-turn trajectories (with LLM-as-Judge scoring), and exposes a rail-v1 batch upload endpoint. It is launched by the online-RL launcher via a uvicorn factory string.
4- 
5-Subpackage layout: `upstream/` (transport and forwarding) → `trajectory/` (persistence and ingestion) → `app/` (FastAPI routes, wiring, CLI/factory). The sole production entry point is the uvicorn factory `openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy:create_app`.
6- 
7-## class openjiuwen.agent_evolving.agent_rl.online.gateway.config.GatewayConfig
8- 
9-```python
10-@dataclass
11-class GatewayConfig(port: int, host: str = "127.0.0.1", llm_url: str = "http://127.0.0.1:18000", judge_url: str = "http://127.0.0.1:18001", model_id: str = "", judge_model: str = "", request_timeout: float = 120.0, llm_api_key: str = "", judge_api_key: str = "", gateway_api_key: str = "", record_dir: str = "records", log_level: str = "INFO", dump_token_ids: bool = False, lora_repo_root: str = "", redis_url: str = "", upstream_max_retries: int = 2, upstream_retry_backoff_sec: float = 0.2, upstream_retry_max_backoff_sec: float = 2.0, disable_gateway_trajectory_collection: bool = False, single_user_default: bool = True)
12-```
13- 
14-Gateway runtime config dataclass.
15- 
16-**Fields**:
17- 
18-* **port**(int): Listen port (required).
19-* **host**(str): Listen address. Default: `"127.0.0.1"`.
20-* **llm_url**(str): Upstream LLM service URL. Default: `"http://127.0.0.1:18000"`.
21-* **judge_url**(str): Judge LLM service URL. Default: `"http://127.0.0.1:18001"`.
22-* **model_id**(str): Model ID. Default: `""`.
23-* **judge_model**(str): Judge model name. Default: `""`.
24-* **request_timeout**(float): Request timeout in seconds. Default: `120.0`.
25-* **llm_api_key**(str): Upstream LLM bearer key. Default: `""`.
26-* **judge_api_key**(str): Judge bearer key. Default: `""`.
27-* **gateway_api_key**(str): Gateway's own auth key. Default: `""`.
28-* **record_dir**(str): Record directory. Default: `"records"`.
29-* **log_level**(str): Log level. Default: `"INFO"`.
30-* **dump_token_ids**(bool): Whether to dump token IDs in logs. Default: `False`.
31-* **lora_repo_root**(str): LoRA repository root. Default: `""`.
32-* **redis_url**(str): Redis URL. Default: `""`.
33-* **upstream_max_retries**(int): Upstream max retries. Default: `2`.
34-* **upstream_retry_backoff_sec**(float): Upstream retry backoff base in seconds. Default: `0.2`.
35-* **upstream_retry_max_backoff_sec**(float): Upstream retry max backoff in seconds. Default: `2.0`.
36-* **disable_gateway_trajectory_collection**(bool): Disable trajectory collection. Default: `False`.
37-* **single_user_default**(bool): Single-user default mode. Default: `True`.
38- 
39-## Constant NON_STANDARD_BODY_KEYS
40- 
41-```python
42-NON_STANDARD_BODY_KEYS: set[str] = {"session_id", "session_done", "turn_type", "memory_scope", "user_id", "workspace_id"}
43-```
44- 
45-Non-standard keys stripped from the body before forwarding.
46- 
47-## def openjiuwen.agent_evolving.agent_rl.online.gateway.common.utc_now_iso
48- 
49-```python
50-def utc_now_iso() -> str
51-```
52- 
53-Returns the current UTC time as an ISO-8601 string.
54- 
55-## def openjiuwen.agent_evolving.agent_rl.online.gateway.common.fit_list
56- 
57-```python
58-def fit_list(values: list[float], expected_len: int) -> list[float]
59-```
60- 
61-Truncates or pads `values` with `0.0` so it has exactly `expected_len` entries; returns `[]` when `expected_len <= 0`.
62- 
63-## def openjiuwen.agent_evolving.agent_rl.online.gateway.message_utils.flatten_message_content
64- 
65-```python
66-def flatten_message_content(content: Any) -> str
67-```
68- 
69-Normalizes message content to a string: `str` returned as-is; `list` joins `text` parts of `{"type": "text"}` items with spaces; `None` returns `""`; otherwise `str(content)`.
70- 
71-## def openjiuwen.agent_evolving.agent_rl.online.gateway.message_utils.extract_last_user_instruction
72- 
73-```python
74-def extract_last_user_instruction(messages: list[dict]) -> str
75-```
76- 
77-Iterates `messages` in reverse, returning the flattened content of the last `role == "user"` message with non-empty text; returns `""` if none.
78- 
79-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.upstream_client.RetryPolicy
80- 
81-```python
82-@dataclass(frozen=True)
83-class RetryPolicy(max_retries: int = 2, backoff_base_sec: float = 0.2, backoff_max_sec: float = 2.0)
84-```
85- 
86-Frozen dataclass for upstream retry policy.
87- 
88-### def backoff_for_attempt
89- 
90-```python
91-def backoff_for_attempt(attempt: int) -> float
92-```
93- 
94-Exponential backoff `backoff_base_sec * 2**(attempt-1)`, clamped to `[0.0, backoff_max_sec]`; returns `0.0` for `attempt <= 0`.
95- 
96-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.upstream_client.UpstreamGatewayClient
97- 
98-```python
99-class UpstreamGatewayClient(Protocol)
100-```
101- 
102-Structurally-typed (`typing.Protocol`) interface for the upstream transport client.
103- 
104-### async def post_chat_completions
105- 
106-```python
107-async def post_chat_completions(*, json_body: dict[str, Any], headers: dict[str, str]) -> httpx.Response
108-```
109- 
110-POST to `/v1/chat/completions`.
111- 
112-### async def request
113- 
114-```python
115-async def request(*, method: str, url: str, params: dict[str, Any], headers: dict[str, str], content: bytes) -> httpx.Response
116-```
117- 
118-Sends an arbitrary request.
119- 
120-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.upstream_client.HTTPXUpstreamGatewayClient
121- 
122-```python
123-class HTTPXUpstreamGatewayClient(*, http_client: httpx.AsyncClient, llm_url: str, retry_policy: RetryPolicy | None = None)
124-```
125- 
126-httpx implementation of `UpstreamGatewayClient`.
127- 
128-**Parameters** (all keyword-only):
129- 
130-* **http_client**(httpx.AsyncClient): Async HTTP client.
131-* **llm_url**(str): Upstream LLM URL.
132-* **retry_policy**(RetryPolicy | None, optional): Retry policy; uses defaults when `None`. Default: `None`.
133- 
134-### async def post_chat_completions
135- 
136-```python
137-async def post_chat_completions(*, json_body: dict[str, Any], headers: dict[str, str]) -> httpx.Response
138-```
139- 
140-POSTs to `{llm_url}/v1/chat/completions` with retry via `_request_with_retry`.
141- 
142-### async def request
143- 
144-```python
145-async def request(*, method: str, url: str, params: dict[str, Any], headers: dict[str, str], content: bytes) -> httpx.Response
146-```
147- 
148-Sends an arbitrary request via `http_client.request(...)` with retry via `_request_with_retry`.
149- 
150-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.forwarder.Forwarder
151- 
152-```python
153-class Forwarder(*, upstream_client: UpstreamGatewayClient, model_id: str)
154-```
155- 
156-LLM request forwarder.
157- 
158-**Parameters** (all keyword-only):
159- 
160-* **upstream_client**(UpstreamGatewayClient): Transport client.
161-* **model_id**(str): Default model ID.
162- 
163-### async def forward
164- 
165-```python
166-async def forward(body: dict[str, Any], headers: dict[str, str]) -> dict[str, Any]
167-```
168- 
169-Cleans the body (removes `NON_STANDARD_BODY_KEYS`, forces `stream=False`, drops `stream_options`, defaults `model`, sets `logprobs=True`/`top_logprobs=1`), calls `post_chat_completions`; raises `HTTPException(502)` on `httpx.HTTPStatusError` (detail = first 500 chars of response text); returns `resp.json()`.
170- 
171-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.sample_recorder.SampleRecorder
172- 
173-```python
174-class SampleRecorder(*, sample_file: str, dump_token_ids: bool = False)
175-```
176- 
177-Lightweight sample counter with optional local JSONL dumps.
178- 
179-**Parameters** (all keyword-only):
180- 
181-* **sample_file**(str): JSONL file path.
182-* **dump_token_ids**(bool): Whether to dump full token IDs. Default: `False`.
183- 
184-### async def record_sample
185- 
186-```python
187-async def record_sample(sample: dict[str, Any]) -> None
188-```
189- 
190-Increments the counter; appends either the full sample (when `dump_token_ids`) or a trimmed version via `_sample_for_log`.
191- 
192-### async def snapshot_stats
193- 
194-```python
195-async def snapshot_stats() -> dict[str, int]
196-```
197- 
198-Returns `{"total_samples": self._total_samples}`.
199- 
200-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.pending_judge_store.PendingJudgeStore
201- 
202-```python
203-class PendingJudgeStore(*, redis: Any, ttl_sec: int = 24 * 3600)
204-```
205- 
206-Redis-backed pending delayed-judge store for rail-v1 samples. Raises `ValueError` when `redis` is `None`.
207- 
208-**Parameters** (all keyword-only):
209- 
210-* **redis**(Any): Redis client.
211-* **ttl_sec**(int): Per-sample TTL in seconds. Default: `86400`.
212- 
213-### async def put
214- 
215-```python
216-async def put(sample: dict[str, Any]) -> None
217-```
218- 
219-Writes the sample JSON to a per-sample key (with TTL) and adds it to a per-session sorted set (scored by creation timestamp).
220- 
221-### async def get_by_session
222- 
223-```python
224-async def get_by_session(session_id: str) -> list[dict[str, Any]]
225-```
226- 
227-Returns all pending samples for a session via `ZRANGE` + `MGET`, decoding bytes.
228- 
229-### async def pop_one
230- 
231-```python
232-async def pop_one(session_id: str, trajectory_id: str, step_index: int) -> Optional[dict[str, Any]]
233-```
234- 
235-Atomically (via pipeline) deletes the sample key and removes it from the session sorted set; returns the decoded sample or `None`.
236- 
237-### async def pop_earliest
238- 
239-```python
240-async def pop_earliest(session_id: str) -> Optional[dict[str, Any]]
241-```
242- 
243-Returns the earliest pending sample (by sorted-set order) via `get_by_session` + `pop_one`.
244- 
245-### async def pop_all
246- 
247-```python
248-async def pop_all(session_id: str) -> list[dict[str, Any]]
249-```
250- 
251-Pops and returns all pending samples for a session.
252- 
253-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.judge_dispatcher.JudgeDispatcher
254- 
255-```python
256-class JudgeDispatcher(*, pending_store: Any, record_sample: Any, judge_scorer: Optional[Any] = None)
257-```
258- 
259-Delayed-judge dispatcher.
260- 
261-**Parameters** (all keyword-only):
262- 
263-* **pending_store**(Any): Pending-sample store.
264-* **record_sample**(Any): `record_sample` coroutine.
265-* **judge_scorer**(Optional[Any], optional): A `JudgeScorer` instance. Default: `None`.
266- 
267-### async def on_prev_feedback
268- 
269-```python
270-async def on_prev_feedback(session_id: str, prev_feedback: Optional[dict[str, Any]]) -> int
271-```
272- 
273-Extracts feedback text; pops the earliest pending sample for the session; finalizes it (tag `"prev_feedback"`); records it; returns 1, else 0.
274- 
275-### async def on_session_done
276- 
277-```python
278-async def on_session_done(session_id: str) -> int
279-```
280- 
281-Pops all pending samples for the session; finalizes each (the last tagged `"session_done"`, others `"session_flush"`); records each; returns the count.
282- 
283-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.rail_ingest.RailBatchIngestor
284- 
285-```python
286-class RailBatchIngestor(*, pending_judge_store: Any, judge_dispatcher: Any, default_user_id: str = "")
287-```
288- 
289-rail-v1 batch upload ingestor.
290- 
291-**Parameters** (all keyword-only):
292- 
293-* **pending_judge_store**(Any): Pending-sample store.
294-* **judge_dispatcher**(Any): Judge dispatcher.
295-* **default_user_id**(str): Default user ID. Default: `""`.
296- 
297-### async def ingest_rail_batch
298- 
299-```python
300-async def ingest_rail_batch(payload: dict[str, Any]) -> dict[str, Any]
301-```
302- 
303-Validates `protocol_version == "rail-v1"`, required `session_id`/`trajectory_id`, and that `samples` is a list; calls `judge_dispatcher.on_prev_feedback(session_id, payload.get("prev_feedback"))`; iterates samples, normalizing each via `_normalize_rail_sample` and putting them in the pending store, counting accepted/rejected (first error captured). Raises `ValueError` if all rejected. If `payload.get("session_done")`, calls `judge_dispatcher.on_session_done(session_id)`. Returns a dict: `protocol_version`, `session_id`, `trajectory_id`, `accepted`, `rejected`, `judged`, `session_flushed`.
304- 
305-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.persistence.GatewayTrajectoryRuntime
306- 
307-```python
308-class GatewayTrajectoryRuntime(config: Any, *, redis: Optional[Any] = None)
309-```
310- 
311-Trajectory persistence and rail-ingest wiring; owns scored-sample persistence. Raises `ValueError` when `redis` is `None`.
312- 
313-**Parameters**:
314- 
315-* **config**(Any): `GatewayConfig` (with `record_dir`, `single_user_default`).
316-* **redis**(Optional[Any], optional): Redis client. Default: `None`.
317- 
318-### @property def store_backend
319- 
320-```python
321-@property
322-def store_backend() -> str
323-```
324- 
325-Returns `type(self._trajectory_store).__name__`.
326- 
327-### @property def rail_ingestor
328- 
329-```python
330-@property
331-def rail_ingestor() -> RailBatchIngestor
332-```
333- 
334-Returns the ingestor; raises `RuntimeError` if uninitialized.
335- 
336-### def set_judge_scorer
337- 
338-```python
339-def set_judge_scorer(judge_scorer: Optional[Any]) -> None
340-```
341- 
342-Rebuilds a `JudgeDispatcher` (with `pending_store`, `record_sample`, `judge_scorer`) and a `RailBatchIngestor` (with `pending_judge_store`, `judge_dispatcher`, `default_user_id`) and stores both.
343- 
344-### async def record_sample
345- 
346-```python
347-async def record_sample(sample: dict[str, Any]) -> None
348-```
349- 
350-Normalizes `user_id` (defaulting to `_default_user_id`, raising `ValueError` if missing); saves to the Redis trajectory store and records via `SampleRecorder`.
351- 
352-### async def snapshot_stats
353- 
354-```python
355-async def snapshot_stats() -> dict[str, Any]
356-```
357- 
358-Returns merged stats: `total_samples`, `trajectory_store_backend`, `trajectory_store_total`, `trajectory_store_pending`, `trajectory_store_training`, `trajectory_store_trained`, `trajectory_store_failed`.
359- 
360-## def openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.sample_payloads.build_sample
361- 
362-```python
363-def build_sample(*, user_id: str, session_id: str, turn_num: int, mode: str, io_mode: str, model: Any, messages: list[dict[str, Any]], tools: Any, assistant_message: dict[str, Any], usage: dict[str, Any], finish_reason: Optional[str], prompt_text: str, prompt_ids: list[int], response_text: str, response_ids: list[int], response_logprobs: list[float], tool_calls: list[dict[str, Any]], request_extras: Optional[dict[str, Any]] = None, sample_id: Optional[str] = None, created_at: Optional[str] = None, extra_fields: Optional[dict[str, Any]] = None) -> dict[str, Any]
364-```
365- 
366-Builds the normalized sample dict with `sample_id` (default uuid4), `created_at` (default `utc_now_iso()`), `user_id`, `session_id`, `turn_num`, `mode`, `io_mode`, `model`, nested `request` (messages, tools, **request_extras), nested `response` (message, usage, finish_reason), and nested `trajectory` (`input_ids = prompt_ids + response_ids`, `attention_mask`, `response_mask`, `prompt_text`, `prompt_ids`, `response_text`, `response_ids`, `response_logprobs`, `tool_calls`). Merges `extra_fields` at the top level if provided.
367- 
368-## def openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.sample_payloads.coerce_logprobs
369- 
370-```python
371-def coerce_logprobs(values: Any, expected_len: int) -> list[float]
372-```
373- 
374-Converts arbitrary logprob values to floats (skipping non-numeric), then `fit_list(out, expected_len)` to a fixed length.
375- 
376-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.bootstrap.build_app_from_config
377- 
378-```python
379-def build_app_from_config(config: GatewayConfig, *, http_client: Any = None, redis_client: Any = None) -> FastAPI
380-```
381- 
382-Production entry that assembles a FastAPI app from a `GatewayConfig`. Configures logging; creates/accepts a Redis async client (requires `redis_url` or an injected `redis_client`, else raises `ValueError`); creates/accepts an `httpx.AsyncClient`; constructs `HTTPXUpstreamGatewayClient` with a `RetryPolicy`; constructs `Forwarder`; constructs `GatewayTrajectoryRuntime(config, redis=redis_client)`; optionally constructs a `JudgeScorer` when `config.judge_url` is set, then `set_judge_scorer(judge_scorer)`; optionally loads `LoRARepository`; defines an inner `async def close_resources()` that closes owned http/redis clients; returns `build_gateway_app(config=..., forwarder=..., upstream_client=..., trajectory_runtime=..., close_resources=..., lora_repo=...)`.
383- 
384-**Parameters**:
385- 
386-* **config**(GatewayConfig): Config.
387-* **http_client**(Any, optional): Injected HTTP client. Default: `None`.
388-* **redis_client**(Any, optional): Injected Redis client. Default: `None`.
389- 
390-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.server.build_gateway_app
391- 
392-```python
393-def build_gateway_app(*, config: Any, forwarder: Forwarder, upstream_client: UpstreamGatewayClient, trajectory_runtime: GatewayTrajectoryRuntime, close_resources: Callable[[], Awaitable[None]], lora_repo: Any = None) -> FastAPI
394-```
395- 
396-Creates a `FastAPI(title="Online-RL Gateway", lifespan=...)` whose lifespan calls `close_resources()` on shutdown. Registers routes:
397- 
398-- `GET /health` → `{"status": "ok"}`
399-- `GET /v1/gateway/stats` → auth-gated; returns `_snapshot_stats(...)` with the current request count (guarded by an `asyncio.Lock`).
400-- `POST /v1/gateway/upload/batch` → auth-gated; calls `trajectory_runtime.rail_ingestor.ingest_rail_batch(payload)`, returns `{"ok": True, "result": result}`; `ValueError` → `HTTPException(400)`.
401-- `POST /v1/chat/completions` → auth-gated; increments counter; parses JSON body; resolves `user_id`; injects latest LoRA; pops `stream`; calls `_forward_chat_completions`; returns `StreamingResponse(stream_chat_response(...))` if the client wanted a stream, else `JSONResponse`.
402-- `GET/POST/PUT/PATCH/DELETE/OPTIONS/HEAD /{path:path}` (catch-all proxy) → auth-gated; forwards to `upstream_client.request(...)` targeting `{config.llm_url}/{path}`, strips hop-by-hop headers, returns `Response`.
403- 
404-**Parameters** (all keyword-only):
405- 
406-* **config**(Any): `GatewayConfig`.
407-* **forwarder**(Forwarder): Forwarder.
408-* **upstream_client**(UpstreamGatewayClient): Upstream client.
409-* **trajectory_runtime**(GatewayTrajectoryRuntime): Trajectory runtime.
410-* **close_resources**(Callable[[], Awaitable[None]]): Close-resources coroutine.
411-* **lora_repo**(Any, optional): LoRA repository. Default: `None`.
412- 
413-## async def openjiuwen.agent_evolving.agent_rl.online.gateway.app.http_helpers.ensure_gateway_auth
414- 
415-```python
416-async def ensure_gateway_auth(gateway_api_key: str, authorization: Optional[str]) -> None
417-```
418- 
419-No-op when `gateway_api_key` is empty; otherwise requires a `Bearer ` token matching the key, raising `HTTPException(401)` if missing and `HTTPException(403)` if invalid.
420- 
421-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.http_helpers.build_upstream_headers
422- 
423-```python
424-def build_upstream_headers(request: Request, *, llm_api_key: str) -> dict[str, str]
425-```
426- 
427-Copies inbound request headers, dropping `host`/`content-length`/`connection` and any `x-forwarded-*`; injects `Authorization: Bearer {llm_api_key}` when `llm_api_key` is set.
428- 
429-## async def openjiuwen.agent_evolving.agent_rl.online.gateway.app.http_helpers.stream_chat_response
430- 
431-```python
432-async def stream_chat_response(response_json: dict[str, Any], *, model_id: str)
433-```
434- 
435-Async generator (yields SSE strings) that wraps a non-streaming chat response into a synthetic SSE stream: emits a first chunk (with delta role/content/tool_calls/reasoning_content, token_ids, logprobs, prompt_token_ids), a final chunk (with `finish_reason` and `usage`), then `data: [DONE]`.
436- 
437-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.request_context.resolve_trace_id
438- 
439-```python
440-def resolve_trace_id(request: Request) -> str
441-```
442- 
443-Returns the `x-request-id` header if present, else a synthesized `uuid.uuid4().hex[:8]`.
444- 
445-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.request_context.require_messages
446- 
447-```python
448-def require_messages(body: dict[str, Any]) -> list[dict[str, Any]]
449-```
450- 
451-Validates that `body["messages"]` is a non-empty list; raises `HTTPException(400)` otherwise.
452- 
453-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.request_context.require_user_id
454- 
455-```python
456-def require_user_id(request: Request, config: Any) -> str
457-```
458- 
459-Reads the `x-user-id` header; if empty and `config.single_user_default` is truthy, falls back to a default ID; raises `HTTPException(400)` if still empty.
460- 
461-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy.create_app
462- 
463-```python
464-def create_app()
465-```
466- 
467-Factory entry for `uvicorn ...gateway.app.proxy:create_app --factory`; builds config from env via `_build_config_from_env()` then returns `build_app_from_config(config)`.
468- 
469-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy.main
470- 
471-```python
472-def main() -> None
473-```
474- 
475-CLI entry point. Parses args (`--host`, `--port` required, `--llm-url`, `--judge-url`, `--model-id`, `--judge-model`, `--record-dir`, `--lora-repo-root`, `--log-level`), builds a `GatewayConfig`, calls `build_app_from_config`, then runs via `uvicorn.run(...)`.
476- 
477-## Usage
478- 
479-- [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py): References the gateway only as a uvicorn factory **string** (`DEFAULT_GATEWAY_APP_FACTORY = 'openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy:create_app'`), spawning `uvicorn ... --factory` as a subprocess. This is the sole production entry point.
480-- [judge_scorer.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/judge/judge_scorer.py): Provides `JudgeScorer`, constructed by `app/bootstrap.py`, whose `score(...)` is called by `JudgeDispatcher._finalize_sample`.
481-- [redis_trajectory_store.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/storage/redis_trajectory_store.py): Provides `RedisTrajectoryStore`, constructed by `trajectory/persistence.py`; `save_sample(...)` and `stats()` are used by `GatewayTrajectoryRuntime`.
482-- [lora_repo.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/storage/lora_repo.py): Provides `LoRARepository`, optionally constructed by `app/bootstrap.py`; `get_latest(user_id)` is called by `app/server.py`'s `_inject_latest_lora`.
483-- Tests: [test_forwarder.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_forwarder.py), [test_processor_components.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_processor_components.py), [test_upstream_client.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_upstream_client.py), [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py), [test_online_gateway_e2e.py](file:///Users/dongdong/Desktop/project/agent-core/tests/system_tests/agent_evolving/agent_rl/online/test_online_gateway_e2e.py).
Ddocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/online/inference.md+0-90
@@ -1,90 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.inference
2- 
3-LoRA hot-loading notifier for the vLLM inference service. By calling vLLM's native `/v1/load_lora_adapter` and `/v1/unload_lora_adapter` endpoints, it hot-loads (or unloads) a LoRA adapter for a specific user without restarting the service. After online RL training completes and publishes a LoRA, the scheduler calls this class to notify vLLM to apply the new weights.
4- 
5-## class openjiuwen.agent_evolving.agent_rl.online.inference.notifier.InferenceNotifier
6- 
7-```python
8-class InferenceNotifier(vllm_base_url: str, timeout: float = 120.0, http_client: Optional[httpx.AsyncClient] = None)
9-```
10- 
11-Asynchronous HTTP client responsible for notifying vLLM to hot-load/unload LoRA adapters.
12- 
13-**Parameters**:
14- 
15-* **vllm_base_url**(str): Base URL of the vLLM service (trailing `/` is stripped), e.g. `http://vllm.local`.
16-* **timeout**(float): Per-request timeout in seconds. Default: `120.0`.
17-* **http_client**(Optional[httpx.AsyncClient], optional): Externally injected async HTTP client. When `None`, the notifier creates and owns its own `httpx.AsyncClient`. Default: `None`.
18- 
19-**Notes**:
20- 
21-- When an external `http_client` is passed in, the notifier does not own the client and `close()` will not close it; a self-created client is owned by the notifier and released on close.
22- 
23-### async def close
24- 
25-```python
26-async def close() -> None
27-```
28- 
29-Closes the underlying HTTP client.
30- 
31-**Notes**:
32- 
33-- Closes the HTTP client only when the notifier created (owns) it; if the client was externally injected, this method is a no-op. Safe to call from an async context.
34- 
35-### async def notify_update
36- 
37-```python
38-async def notify_update(user_id: str, lora_path: str) -> None
39-```
40- 
41-Notifies vLLM to hot-load the LoRA adapter for the specified user.
42- 
43-**Parameters**:
44- 
45-* **user_id**(str): User identifier, also used as vLLM's `lora_name`. After loading, requests specifying this `lora_name` automatically apply the new weights.
46-* **lora_path**(str): Absolute path to the LoRA weights directory.
47- 
48-**Notes**:
49- 
50-- Sends an HTTP POST to `{vllm_base_url}/v1/load_lora_adapter` with JSON body:
51- 
52-```json
53-{
54- "lora_name": "<user_id>",
55- "lora_path": "<lora_path>",
56- "load_inplace": true
57-}
58-```
59- 
60-- Uses `self.timeout` as the request timeout.
61-- When the HTTP status code is `>= 400`, raises `RuntimeError` with message like `vLLM load_lora_adapter failed: status=<code>, body=<first 400 chars of body>`.
62-- On success, logs at INFO: `LoRA hot-loaded for user %s: %s`.
63- 
64-### async def unload
65- 
66-```python
67-async def unload(user_id: str) -> None
68-```
69- 
70-Unloads the LoRA adapter for the specified user (useful for cleaning up inactive users).
71- 
72-**Parameters**:
73- 
74-* **user_id**(str): The `lora_name` (user identifier) to unload.
75- 
76-**Notes**:
77- 
78-- Sends an HTTP POST to `{vllm_base_url}/v1/unload_lora_adapter` with JSON body `{"lora_name": "<user_id>"}`.
79-- Raises HTTP errors via `resp.raise_for_status()` (unlike `notify_update`'s custom `RuntimeError`).
80-- On success, logs at INFO: `LoRA unloaded for user %s`.
81- 
82-## Usage
83- 
84-`InferenceNotifier` is constructed or called in the following locations:
85- 
86-- [ppo_executor.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/scheduler/ppo_executor.py): `PPOTrainingExecutor` receives `Optional[InferenceNotifier]` in its constructor; `aclose()` calls `close()`; after a successful LoRA publish, calls `notify_update(user_id, published_lora_path)` (failures are treated as non-fatal warnings).
87-- [online_training_scheduler.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/scheduler/online_training_scheduler.py): Forwards `notifier` when constructing `PPOTrainingExecutor`.
88-- [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py): `start_online_training_scheduler` constructs `InferenceNotifier(runtime.inference_url)` and passes it to the scheduler.
89-- [rl_optimizer.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/optimizer/rl_optimizer.py): `setup_inference(vllm_url)` stores the URL; at build time constructs `InferenceNotifier` and passes it to `OnlineTrainingScheduler`.
90-- [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py): Test injects a fake `httpx.AsyncClient` to verify `notify_update`'s request path and body, and verifies that an external client is not closed by `close()`.
Mdocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/online/judge.md+8-10
@@ -1,6 +1,6 @@
1# openjiuwen.agent_evolving.agent_rl.online.judge1# openjiuwen.agent_evolving.agent_rl.online.judge
2 2 
3-LLM-as-a-Judge reward scorer. Scores agent turns along four dimensions (task completion, response quality, tool usage, coherence, each 0-10), normalizes the aggregate to a `[-1, 1]` reward, and supports multi-vote averaging with retry logic. It is the reward source consumed by the gateway's trajectory pipeline (the "delayed judge" flow).3+LLM-as-a-Judge reward scorer. Scores agent turns along four dimensions (task completion, response quality, tool usage, coherence, each 0-10), normalizes the aggregate to a `[0, 1]` reward, and supports multi-vote averaging with retry logic. It is the reward source consumed by the RL Service capture pipeline (the "delayed judge" flow).
4 4 
5The package is layered in three tiers: `scoring.py` (pure prompt/parse/normalize, no I/O) → `evaluator.py` (async HTTP voting engine) → `judge_scorer.py` (high-level client adapter); `judge_server.py` is an optional standalone FastAPI service.5The package is layered in three tiers: `scoring.py` (pure prompt/parse/normalize, no I/O) → `evaluator.py` (async HTTP voting engine) → `judge_scorer.py` (high-level client adapter); `judge_server.py` is an optional standalone FastAPI service.
6 6 
@@ -53,7 +53,7 @@ Robustly extracts the judge's JSON score dict from a model response. First tries
53def normalize_overall_score(overall: float) -> float53def normalize_overall_score(overall: float) -> float
54```54```
55 55 
56-Maps a raw 0-10 score to `[-1, 1]` via `(overall - 5.0) / 5.0`.56+Maps a raw 0-10 score to `[0, 1]` via `overall / 10.0`.
57 57 
58**Parameters**:58**Parameters**:
59 59 
@@ -61,7 +61,7 @@ Maps a raw 0-10 score to `[-1, 1]` via `(overall - 5.0) / 5.0`.
61 61 
62**Returns**:62**Returns**:
63 63 
64-`float`, the normalized `[-1, 1]` reward.64+`float`, the normalized `[0, 1]` reward.
65 65 
66## class openjiuwen.agent_evolving.agent_rl.online.judge.evaluator.JudgeEvaluatorConfig66## class openjiuwen.agent_evolving.agent_rl.online.judge.evaluator.JudgeEvaluatorConfig
67 67 
@@ -108,7 +108,7 @@ Core scoring entry. Builds messages, runs `config.num_votes` parallel `_query_vo
108 108 
109```python109```python
110{110{
111- "score": float, # normalized [-1, 1]111+ "score": float, # normalized [0, 1]
112 "overall_raw": float, # averaged raw 0-10 score112 "overall_raw": float, # averaged raw 0-10 score
113 "votes": list[float], # per-vote overall values113 "votes": list[float], # per-vote overall values
114 "details": dict | list, # single vote dict if num_votes==1 else list114 "details": dict | list, # single vote dict if num_votes==1 else list
@@ -249,9 +249,7 @@ Entry point. Configures logging, constructs `JudgeConfig`, `create_app`, `uvicor
249 249 
250## Usage250## Usage
251 251 
252-- [bootstrap.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/app/bootstrap.py): Constructs `JudgeScorer` when `config.judge_url` is set and injects it via `trajectory_runtime.set_judge_scorer(judge_scorer)`. This is the sole production construction site for `JudgeScorer`.252+- `service.py` constructs `JudgeScorer` from the RL Service configuration and injects it into `CapturePipeline`.
253-- [persistence.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/trajectory/persistence.py): `set_judge_scorer` passes it into a new `JudgeDispatcher`.253+- `capture_pipeline.py` calls `JudgeScorer.score(...)` while atomically publishing a completed delayed-feedback turn.
254-- [judge_dispatcher.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/trajectory/judge_dispatcher.py): `_finalize_sample` calls `await self._judge_scorer.score(...)` (exceptions fall back to `{"score": 0.0, ...}`). This is the sole production call site of `JudgeScorer.score`.254+- `test_gateway_support.py` covers score parsing and the `finish_reason=="length"` retry path.
255-- [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py): Tests `JudgeScorer._parse_scores` and the `finish_reason=="length"` retry path.255+- `test_capture_pipeline.py` covers the scorer interface, atomic publication, and Judge failure handling.
256-- [test_processor_components.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_processor_components.py): Defines a duck-typed `_FakeJudgeScorer` verifying the `.score(...)` interface.
257-- Note: `judge_server.py`'s `create_app`/`main`/`JudgeConfig`/`ScoreRequest`/`ScoreResponse` are **not imported by any module**. The launcher starts the judge as a raw vLLM process, not this FastAPI service; this module is an optional standalone service (`python -m judge.judge_server ...`).
Ddocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/online/launcher.md+0-369
@@ -1,369 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.launcher
2- 
3-Orchestration runtime for the JiuwenClaw online RL loop (interact → trajectory collect → PPO train → LoRA hot-load). Parses CLI args, merges config, spawns service processes (vLLM inference, judge, gateway, training scheduler, JiuwenClaw app and web), and provides graceful shutdown via signal handling and health checks.
4- 
5-The module is clearly layered: `cli.py` parses args, `loader.py` merges config, `services.py` spawns processes, `workspace.py` writes env files, `runner.py` orchestrates the loop, and `__init__.py` re-exports the top-level entry points.
6- 
7-## class openjiuwen.agent_evolving.agent_rl.online.launcher.runner.LauncherPaths
8- 
9-```python
10-@dataclass(frozen=True)
11-class LauncherPaths(agent_core_root: Path, jiuwenclaw_repo: Path, workspace_root: Path, workspace_env: Path, script_dir: Path)
12-```
13- 
14-Frozen dataclass describing the on-disk layout consumed by the runner.
15- 
16-**Fields**:
17- 
18-* **agent_core_root**(Path): agent-core repository root.
19-* **jiuwenclaw_repo**(Path): JiuwenClaw repository root.
20-* **workspace_root**(Path): Workspace root directory.
21-* **workspace_env**(Path): Workspace `.env` file path.
22-* **script_dir**(Path): Script directory (logs dir is under it).
23- 
24-## def openjiuwen.agent_evolving.agent_rl.online.launcher.runner.run_online_rl_loop
25- 
26-```python
27-def run_online_rl_loop(*, cfg: OnlineRLConfig, cfg_path: Path, paths: LauncherPaths) -> None
28-```
29- 
30-Top-level orchestration entry. Installs SIGINT/SIGTERM handlers (raising `_ShutdownRequested`), executes the launch sequence:
31- 
32-1. Creates `logs/` directory.
33-2. `resolve_launch_runtime(cfg, script_dir=paths.script_dir)`.
34-3. Pre-checks required ports.
35-4. Starts inference vLLM (`enable_runtime_lora=True`) unless `runtime.skip_vllm`.
36-5. Starts judge vLLM (`enable_runtime_lora=False`) unless `runtime.skip_judge`.
37-6. Health-checks vLLM and judge.
38-7. Starts gateway, health-checks gateway.
39-8. Starts the training scheduler via `start_online_training_scheduler`.
40-9. If `cfg.jiuwen.enabled`: `ensure_workspace` then `start_jiuwenclaw`; otherwise skips.
41-10. `print_launch_summary`.
42-11. Supervisory loop: every 30s polls each child `Popen.poll()`; if any exited, stops everything and returns.
43-12. `finally` block calls `_shutdown()` which stops the scheduler and terminates web→claw→gateway→judge→vllm in order (idempotent).
44- 
45-**Parameters** (all keyword-only):
46- 
47-* **cfg**(OnlineRLConfig): Run configuration.
48-* **cfg_path**(Path): Config file path.
49-* **paths**(LauncherPaths): Path layout.
50- 
51-## def openjiuwen.agent_evolving.agent_rl.online.launcher.cli.build_arg_parser
52- 
53-```python
54-def build_arg_parser() -> argparse.ArgumentParser
55-```
56- 
57-Constructs the CLI argument parser, described as `'JiuwenClaw online RL loop: interact -> trajectory collect -> PPO train -> LoRA hot-load'`.
58- 
59-**Returns**:
60- 
61-`argparse.ArgumentParser`. Main args (dest / type / default):
62- 
63-| Flag | Dest | Type | Default |
64-|------|------|------|---------|
65-| `--config` | `config` | str | None |
66-| `--model-path` | `model_path` | str | None |
67-| `--model-name` | `model_name` | str | None |
68-| `--vllm-gpu` | `vllm_gpu` | str | None |
69-| `--vllm-tp` | `vllm_tp` | int | None |
70-| `--vllm-port` | `vllm_port` | int | None |
71-| `--judge-model-path` | `judge_model_path` | str | None |
72-| `--judge-model-name` | `judge_model_name` | str | None |
73-| `--judge-gpu` | `judge_gpu` | str | None |
74-| `--judge-tp` | `judge_tp` | int | None |
75-| `--judge-port` | `judge_port` | int | None |
76-| `--gateway-port` | `gateway_port` | int | None |
77-| `--redis-url` | `redis_url` | str | None |
78-| `--threshold` | `threshold` | int | None |
79-| `--scan-interval` | `scan_interval` | int | None |
80-| `--train-gpu` | `train_gpu` | str | None |
81-| `--ppo-config` | `ppo_config` | str | None |
82-| `--trajectory-batch-size` | `trajectory_batch_size` | int | None |
83-| `--lora-repo` | `lora_repo` | str | None |
84-| `--jiuwen-agent-server-port` | `jiuwen_agent_server_port` | int | None |
85-| `--demo` | `demo` | store_true | None |
86-| `--inference-url` | `inference_url` | str | None |
87-| `--judge-url` | `judge_url` | str | None |
88-| `--skip-jiuwen`/`--skip_jiuwen` | `skip_jiuwen` | store_true | False |
89-| `--jiuwen-ws-port` | `jiuwen_ws_port` | int | None |
90-| `--jiuwen-web-host` | `jiuwen_web_host` | str | None |
91-| `--jiuwen-web-port` | `jiuwen_web_port` | int | None |
92- 
93-## def openjiuwen.agent_evolving.agent_rl.online.launcher.cli.build_cli_overrides
94- 
95-```python
96-def build_cli_overrides(args: argparse.Namespace) -> dict[str, object]
97-```
98- 
99-Translates a parsed `Namespace` into a nested override dict. Uses a hardcoded `cli_mappings` table mapping CLI attribute names to dotted config paths (e.g. `model_path` → `inference.model_path`), skipping `None` values. When `--skip-jiuwen` is true, sets `jiuwen.enabled` to `False`.
100- 
101-**Parameters**:
102- 
103-* **args**(argparse.Namespace): Parsed args.
104- 
105-**Returns**:
106- 
107-`dict[str, object]`, a nested override dict.
108- 
109-## def openjiuwen.agent_evolving.agent_rl.online.launcher.loader.load_runtime_config
110- 
111-```python
112-def load_runtime_config(*, config_path: str | None, cli_overrides: dict[str, object]) -> tuple[OnlineRLConfig, Path]
113-```
114- 
115-Three-layer config merge (OmegaConf): built-in `BUILTIN_ONLINE_RL_CONFIG` + optional YAML (falls back to built-in `online_config.py` when `config_path` is missing) + CLI overrides. Returns a Pydantic-validated `OnlineRLConfig` and the resolved path.
116- 
117-**Parameters** (all keyword-only):
118- 
119-* **config_path**(str | None): User YAML path; when `None`, uses built-in defaults.
120-* **cli_overrides**(dict[str, object]): CLI override dict.
121- 
122-**Returns**:
123- 
124-`tuple[OnlineRLConfig, Path]`, the config object and the resolved config file path.
125- 
126-## def openjiuwen.agent_evolving.agent_rl.online.launcher.loader.resolve_builtin_online_config_path
127- 
128-```python
129-def resolve_builtin_online_config_path() -> Path
130-```
131- 
132-Locates the on-disk path of `online_config.py` via the module's `__file__` attribute.
133- 
134-**Returns**:
135- 
136-`Path`, the built-in config file path. Raises `RuntimeError` if it cannot be located.
137- 
138-## Constant DEFAULT_CONFIG_FILENAME
139- 
140-```python
141-DEFAULT_CONFIG_FILENAME = "online_config.py (built-in)"
142-```
143- 
144-Used in CLI help text to indicate the config source.
145- 
146-## class openjiuwen.agent_evolving.agent_rl.online.launcher.services.LaunchRuntime
147- 
148-```python
149-@dataclass(frozen=True)
150-class LaunchRuntime(inference_url: str, judge_url: str, gateway_base_url: str, gateway_api_url: str, lora_repo: str, skip_vllm: bool, skip_judge: bool, reuse_inference_for_judge: bool, judge_label: str, ports_to_check: tuple[tuple[str, str, int], ...])
151-```
152- 
153-Frozen dataclass holding resolved URLs/flags consumed by the runner. Each entry in `ports_to_check` is `(name, host, port)`.
154- 
155-**Fields**:
156- 
157-* **inference_url**(str): Inference service URL.
158-* **judge_url**(str): Judge service URL.
159-* **gateway_base_url**(str): Gateway base URL.
160-* **gateway_api_url**(str): Gateway API URL.
161-* **lora_repo**(str): LoRA repository root.
162-* **skip_vllm**(bool): Whether to skip vLLM launch.
163-* **skip_judge**(bool): Whether to skip judge launch.
164-* **reuse_inference_for_judge**(bool): Whether to reuse the inference service as judge.
165-* **judge_label**(str): Judge label.
166-* **ports_to_check**(tuple[tuple[str, str, int], ...]): Ports to pre-check.
167- 
168-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.resolve_launch_runtime
169- 
170-```python
171-def resolve_launch_runtime(cfg: OnlineRLConfig, *, script_dir: Path) -> LaunchRuntime
172-```
173- 
174-Resolves service URLs, skip flags, port-check list, and LoRA repo location from config.
175- 
176-**Parameters**:
177- 
178-* **cfg**(OnlineRLConfig): Run configuration.
179-* **script_dir**(Path): Script directory (used to resolve the LoRA repo relative path).
180- 
181-**Returns**:
182- 
183-`LaunchRuntime`.
184- 
185-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.url_host
186- 
187-```python
188-def url_host(host: str) -> str
189-```
190- 
191-Normalizes wildcard bind hosts (`'0.0.0.0'`, `'::'`) to `'127.0.0.1'` for client-side URL construction.
192- 
193-**Parameters**:
194- 
195-* **host**(str): Bind host.
196- 
197-**Returns**:
198- 
199-`str`, the normalized host.
200- 
201-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.spawn_process
202- 
203-```python
204-def spawn_process(cmd: list[str], *, env: dict[str, str] | None = None, cwd: str | None = None, log_path: Path | None = None) -> subprocess.Popen
205-```
206- 
207-Generic child-process spawner. When `log_path` is provided, appends stdout+stderr to that file (auto-creating parent dirs); otherwise inherits parent stdio.
208- 
209-**Parameters**:
210- 
211-* **cmd**(list[str]): Command list.
212-* **env**(dict[str, str] | None, optional): Environment variables. Default: `None`.
213-* **cwd**(str | None, optional): Working directory. Default: `None`.
214-* **log_path**(Path | None, optional): Log file path. Default: `None`.
215- 
216-**Returns**:
217- 
218-`subprocess.Popen`.
219- 
220-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_vllm_service
221- 
222-```python
223-def start_vllm_service(service_cfg: VLLMServiceConfig, *, step_label: str, service_name: str, enable_runtime_lora: bool, log_path: Path | None = None) -> subprocess.Popen
224-```
225- 
226-Launches a vLLM OpenAI API server (`python -m vllm.entrypoints.openai.api_server`) with args from `service_cfg`. Sets `CUDA_VISIBLE_DEVICES`; when `enable_runtime_lora=True`, sets `VLLM_ALLOW_RUNTIME_LORA_UPDATING=1`.
227- 
228-**Parameters** (all keyword-only except `service_cfg`):
229- 
230-* **service_cfg**(VLLMServiceConfig): Service config.
231-* **step_label**(str): Step label (for logs).
232-* **service_name**(str): Service name (for logs).
233-* **enable_runtime_lora**(bool): Whether to enable runtime LoRA updates.
234-* **log_path**(Path | None, optional): Log path. Default: `None`.
235- 
236-**Returns**:
237- 
238-`subprocess.Popen`.
239- 
240-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_gateway
241- 
242-```python
243-def start_gateway(*, inference_url: str, judge_url: str, judge_model: str, model_id: str, model_path: str, lora_repo_root: str, gateway_cfg: GatewayServiceConfig, agent_core_root: Path, log_path: Path | None = None) -> subprocess.Popen
244-```
245- 
246-Launches the agent-core gateway via `python -m uvicorn <DEFAULT_GATEWAY_APP_FACTORY> --factory ...`. Sets `LLM_URL`, `JUDGE_URL`, `JUDGE_MODEL`, `MODEL_ID`, `MODEL_PATH`, `GATEWAY_HOST/PORT`, `RECORD_DIR`, `REDIS_URL`, optional `LORA_REPO_ROOT`, optional `DISABLE_GATEWAY_TRAJECTORY_COLLECTION` and other env vars, running with `cwd=agent_core_root`.
247- 
248-**Parameters** (all keyword-only):
249- 
250-* **inference_url**(str): Inference URL.
251-* **judge_url**(str): Judge URL.
252-* **judge_model**(str): Judge model name.
253-* **model_id**(str): Model ID.
254-* **model_path**(str): Model path.
255-* **lora_repo_root**(str): LoRA repository root.
256-* **gateway_cfg**(GatewayServiceConfig): Gateway service config.
257-* **agent_core_root**(Path): agent-core root directory.
258-* **log_path**(Path | None, optional): Log path. Default: `None`.
259- 
260-**Returns**:
261- 
262-`subprocess.Popen`.
263- 
264-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_online_training_scheduler
265- 
266-```python
267-def start_online_training_scheduler(*, cfg: OnlineRLConfig, runtime: LaunchRuntime)
268-```
269- 
270-Lazily imports `InferenceNotifier`, `OnlineTrainingScheduler`, `LoRARepository`, constructs and starts the scheduler (polls `RedisTrajectoryStore` and triggers PPO LoRA training when pending trajectories reach `threshold`). Returns the scheduler instance, calling `scheduler.start()` before returning.
271- 
272-**Parameters** (all keyword-only):
273- 
274-* **cfg**(OnlineRLConfig): Run configuration.
275-* **runtime**(LaunchRuntime): Resolved runtime info.
276- 
277-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_jiuwenclaw
278- 
279-```python
280-def start_jiuwenclaw(*, jiuwenclaw_repo: Path, workspace_root: Path, trajectory_gateway_url: str, model_path: str, trajectory_mode: str, trajectory_batch_size: int, app_host: str, ws_port: int, web_host: str, web_port: int) -> tuple[subprocess.Popen, subprocess.Popen | None]
281-```
282- 
283-Launches the JiuwenClaw app (`python -m jiuwenclaw.app`) and optionally its web frontend (`python -m jiuwenclaw.app_web`) when a `web/dist` directory exists. Resolves `RL_ONLINE_TENANT_ID` from env (falling back to `WEB_USER_ID` or `'local-web-user'`), injects `WEB_USER_ID` and a JSON-encoded `CUSTOM_HEADERS` (`{'x-user-id': ...}`). Uses `build_trajectory_env_updates` for trajectory env vars.
284- 
285-**Parameters** (all keyword-only):
286- 
287-* **jiuwenclaw_repo**(Path): JiuwenClaw repository root.
288-* **workspace_root**(Path): Workspace root.
289-* **trajectory_gateway_url**(str): Trajectory gateway URL.
290-* **model_path**(str): Model path.
291-* **trajectory_mode**(str): Trajectory mode.
292-* **trajectory_batch_size**(int): Trajectory batch size.
293-* **app_host**(str): App listen address.
294-* **ws_port**(int): WebSocket port.
295-* **web_host**(str): Web frontend listen address.
296-* **web_port**(int): Web frontend port.
297- 
298-**Returns**:
299- 
300-`tuple[subprocess.Popen, subprocess.Popen | None]`, the app process and optional web process.
301- 
302-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.print_launch_summary
303- 
304-```python
305-def print_launch_summary(*, cfg: OnlineRLConfig, cfg_path: Path, runtime: LaunchRuntime, web_started: bool) -> None
306-```
307- 
308-Logs a formatted summary banner (config path, web/WS URLs, vLLM inference/judge URLs, gateway, Redis, trajectory mode/log, LoRA repo, train threshold, batch size, scan interval, train GPUs, usage hint). The web frontend line is conditionally included based on `cfg.jiuwen.enabled` and `web_started`.
309- 
310-**Parameters** (all keyword-only):
311- 
312-* **cfg**(OnlineRLConfig): Run configuration.
313-* **cfg_path**(Path): Config file path.
314-* **runtime**(LaunchRuntime): Runtime info.
315-* **web_started**(bool): Whether the web frontend was started.
316- 
317-## Constants
318- 
319-```python
320-DEFAULT_GATEWAY_APP_FACTORY = 'openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy:create_app'
321-EXISTING_SERVICE_HEALTH_TIMEOUT = 30.0
322-```
323- 
324-- `DEFAULT_GATEWAY_APP_FACTORY`: The uvicorn app factory target string.
325-- `EXISTING_SERVICE_HEALTH_TIMEOUT`: Health-check timeout in seconds when reusing an externally-managed service.
326- 
327-## def openjiuwen.agent_evolving.agent_rl.online.launcher.workspace.build_trajectory_env_updates
328- 
329-```python
330-def build_trajectory_env_updates(*, gateway_url: str, model_path: str, trajectory_batch_size: int, trajectory_mode: str, trajectory_tenant_id: str | None = None) -> dict[str, str]
331-```
332- 
333-Returns a dict of env vars for JiuwenClaw Rail (online-RL trajectory upload).
334- 
335-**Parameters** (all keyword-only):
336- 
337-* **gateway_url**(str): Gateway URL.
338-* **model_path**(str): Model path.
339-* **trajectory_batch_size**(int): Trajectory batch size.
340-* **trajectory_mode**(str): Trajectory mode.
341-* **trajectory_tenant_id**(str | None, optional): Tenant ID. Default: `None`.
342- 
343-**Returns**:
344- 
345-`dict[str, str]`, containing `USE_RL_ONLINE_RAIL='1'`, `ENABLE_TRAJECTORY_COLLECTION='false'`, `TRAJECTORY_GATEWAY_URL`, `TRAJECTORY_TOKENIZER_PATH`, `TRAJECTORY_BATCH_SIZE`, `TRAJECTORY_MODE`, and optionally `RL_ONLINE_TENANT_ID`.
346- 
347-## def openjiuwen.agent_evolving.agent_rl.online.launcher.workspace.ensure_workspace
348- 
349-```python
350-def ensure_workspace(*, config_env: Path, gateway_url: str, model_name: str, model_path: str, trajectory_mode: str, trajectory_gateway_url: str | None = None, trajectory_batch_size: int = 8) -> None
351-```
352- 
353-Ensures the JiuwenClaw `.env` file at `config_env` points to the gateway. If the file does not exist, lazily imports and calls `jiuwenclaw.utils.prepare_workspace(overwrite=False, preferred_language='zh')`; then merges (preserving existing keys) a set of values: `API_BASE`, `API_KEY='EMPTY'`, `MODEL_NAME`, `MODEL_PROVIDER='OpenAI'`, `WEB_USER_ID`, `CUSTOM_HEADERS` (JSON), `EMBED_*`, `BROWSER_RUNTIME_MCP_ENABLED='0'`, `EVOLUTION_AUTO_SCAN='false'`, plus trajectory updates from `build_trajectory_env_updates`. Writes back the full file.
354- 
355-**Parameters** (all keyword-only):
356- 
357-* **config_env**(Path): `.env` file path.
358-* **gateway_url**(str): Gateway URL.
359-* **model_name**(str): Model name.
360-* **model_path**(str): Model path.
361-* **trajectory_mode**(str): Trajectory mode.
362-* **trajectory_gateway_url**(str | None, optional): Trajectory gateway URL; when `None`, uses `gateway_url`. Default: `None`.
363-* **trajectory_batch_size**(int): Trajectory batch size. Default: `8`.
364- 
365-## Usage
366- 
367-- [run_online_rl.py](file:///Users/dongdong/Desktop/project/agent-core/examples/jiuwenrl_online/run_online_rl.py): The user-facing launcher script. Imports `build_arg_parser`, `build_cli_overrides`, `load_runtime_config`, `LauncherPaths`, `run_online_rl_loop`, constructs a `LauncherPaths`, then calls the main entry. This is the sole production external consumer.
368-- [test_launcher_runner.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_launcher_runner.py): Tests `run_online_rl_loop` signal shutdown, `print_launch_summary`, `ensure_workspace`, `start_jiuwenclaw`.
369-- [README.md](file:///Users/dongdong/Desktop/project/agent-core/examples/jiuwenrl_online/README.md): References `print_launch_summary` output.
Mdocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/online/rail.md+4-6
@@ -304,9 +304,7 @@ Environment variables consumed:
304 304 
305## Usage305## Usage
306 306 
307-- [\_\_init\_\_.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/__init__.py): Lazy-exports `RLOnlineRail` via `__getattr__` (canonical external import path is `openjiuwen.agent_evolving.agent_rl.RLOnlineRail`) and lists it in `__all__`.307+- `openjiuwen.agent_evolving.agent_rl.__init__` lazy-exports `RLOnlineRail` as the canonical external import.
308-- [workspace.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/workspace.py) and [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py): Set `USE_RL_ONLINE_RAIL=1`, `TRAJECTORY_GATEWAY_URL`, `RL_ONLINE_TENANT_ID` and other env vars for the spawned JiuwenClaw process (do not import rail classes directly).308+- AIGW owns the public `POST /v1/gateway/upload/batch` endpoint and proxies accepted batches to the running RL Service.
309-- [server.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/app/server.py): Registers the `POST /v1/gateway/upload/batch` endpoint that receives uploads; consumed by the `RailBatchIngestor` in [rail_ingest.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/trajectory/rail_ingest.py).309+- The Agent lifecycle is external. Applications that opt into `RLOnlineRail` configure it directly; the RL Service does not spawn an Agent.
310-- [test_rl_online_rail.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_rl_online_rail.py): Imports and constructs `RLOnlineRail`.310+- `test_rl_online_rail.py` covers `RLOnlineRail`, `TrajectoryUploader`, and `OnlineTrajectoryConverter`.
311-- [test_online_gateway_e2e.py](file:///Users/dongdong/Desktop/project/agent-core/tests/system_tests/agent_evolving/agent_rl/online/test_online_gateway_e2e.py): Dynamically imports `RLOnlineRail` and `TrajectoryUploader` via `importlib`.
312-- [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py): Imports `OnlineTrajectoryConverter` and tests `convert` / `to_dict`.
Ddocs/en/2.Development Guide/API Docs/openjiuwen.agent_evolving/agent_rl/online/scheduler.md+0-184
@@ -1,184 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.scheduler
2- 
3-Online RL training polling scheduler and PPO batch executor. A background thread polls the Redis trajectory store for accumulated samples; when a user's sample count crosses a threshold, it triggers a PPO LoRA training batch: convert samples → call Ray/verl training → export LoRA → publish to repository → notify vLLM to hot-load.
4- 
5-## class openjiuwen.agent_evolving.agent_rl.online.scheduler.online_training_scheduler.OnlineTrainingScheduler
6- 
7-```python
8-class OnlineTrainingScheduler(*, redis_url: str = "redis://127.0.0.1:6379/0", poll_interval: float = 30.0, min_samples_for_training: int = 32, base_model_path: str = "", lora_repo: Optional[LoRARepository] = None, notifier: Optional[InferenceNotifier] = None, nproc_per_node: int = 1, training_gpu_ids: str = "", tmp_root: str = "/tmp/agent_rl_online", ppo_config_path: Optional[str] = None)
9-```
10- 
11-Background-thread scheduler. Polls `RedisTrajectoryStore` and starts an asyncio task running a PPO training batch for users whose sample count exceeds the threshold; retains at most one in-flight training task at a time.
12- 
13-**Parameters** (all keyword-only):
14- 
15-* **redis_url**(str): Redis connection URL. Default: `"redis://127.0.0.1:6379/0"`. An empty string disables the scheduler (used in tests).
16-* **poll_interval**(float): Polling interval in seconds. Default: `30.0`.
17-* **min_samples_for_training**(int): Minimum sample count threshold to trigger training. Default: `32`.
18-* **base_model_path**(str): Base model path. Default: `""`.
19-* **lora_repo**(Optional[LoRARepository], optional): LoRA repository instance for publishing training artifacts. Default: `None`.
20-* **notifier**(Optional[InferenceNotifier], optional): vLLM hot-load notifier. Default: `None`.
21-* **nproc_per_node**(int): Number of GPUs per node. Default: `1`.
22-* **training_gpu_ids**(str): Comma-separated GPU IDs for training. Default: `""`.
23-* **tmp_root**(str): Training temp root directory. Default: `"/tmp/agent_rl_online"`.
24-* **ppo_config_path**(Optional[str], optional): Custom Hydra PPO YAML path. Default: `None`.
25- 
26-**Notes**:
27- 
28-- At construction, internally creates a `PPOTrainingExecutor` passing `base_model_path`, `lora_repo`, `notifier`, `nproc_per_node`, `training_gpu_ids`, `ppo_config_path`.
29- 
30-### def start
31- 
32-```python
33-def start() -> None
34-```
35- 
36-Starts the daemon polling thread (thread name `OnlineTrainScheduler`, target `_poll_loop`). If already running, logs a warning and does nothing.
37- 
38-### def stop
39- 
40-```python
41-def stop() -> None
42-```
43- 
44-Signals stop, joins the thread (15s timeout), then calls `self._trainer.close()`. Logs a warning if the thread is still alive. Idempotent.
45- 
46-### async def _poll_loop (private)
47- 
48-```python
49-async def _poll_loop() -> None
50-```
51- 
52-Background thread entry. Creates a dedicated asyncio event loop, lazily imports `redis.asyncio.from_url`, builds `RedisTrajectoryStore`, runs `_poll_main`; in `finally` closes the trainer, Redis client, and loop. Returns immediately when `redis_url` is empty.
53- 
54-### async def _poll_main (private)
55- 
56-```python
57-async def _poll_main() -> None
58-```
59- 
60-Main loop: while not stopped — `_reap_training_task()` → `_poll_once()` → `sleep(poll_interval)`; after the loop, `await _reap_training_task(wait=True)` to drain in-flight tasks. Per-iteration exceptions are caught and logged.
61- 
62-### async def _poll_once (private)
63- 
64-```python
65-async def _poll_once() -> None
66-```
67- 
68-No-op when the store is `None` or a training task is already active. Otherwise calls `get_users_above_threshold(min_samples_for_training)`; for the first user with fetchable samples calls `fetch_and_mark_training(user_id, min_samples_for_training)`, creates `asyncio.create_task(self._train_batch(...))`, starting at most one training task per cycle.
69- 
70-### async def _reap_training_task (private)
71- 
72-```python
73-async def _reap_training_task(*, wait: bool = False) -> None
74-```
75- 
76-Reaps the in-flight training task. Returns immediately if no task; unless `wait=True`, returns early if the task is not done. Otherwise awaits the task; exceptions are logged; `_active_training_task` and `_active_training_user` are cleared in `finally`.
77- 
78-### async def _train_batch (private)
79- 
80-```python
81-async def _train_batch(*, user_id: str, samples: list[dict[str, Any]], sample_ids: list[str]) -> None
82-```
83- 
84-Executes a single training batch. Calls `self._trainer.train_batch(user_id=..., samples=..., training_count=self._training_count, tmp_root=self.tmp_root)`; on success calls `mark_trained(sample_ids)`, on exception calls `mark_failed(sample_ids)`. No-op when the store is `None`.
85- 
86-## def openjiuwen.agent_evolving.agent_rl.online.scheduler.ppo_config.compose_online_ppo_config
87- 
88-```python
89-def compose_online_ppo_config(*, model_path: str, n_gpus_per_node: int = 2, config_path: Optional[str] = None)
90-```
91- 
92-Composes the Hydra/OmegaConf config for online PPO training.
93- 
94-**Parameters** (all keyword-only):
95- 
96-* **model_path**(str): Base model path, written to `cfg.actor_rollout_ref.model.path`.
97-* **n_gpus_per_node**(int): Number of GPUs per node, written to `cfg.trainer.n_gpus_per_node`. Default: `2`.
98-* **config_path**(Optional[str], optional): User-defined Hydra YAML path. When `None`, loads verl's built-in `ppo_trainer` config and merges `ONLINE_PPO_VERL_HYDRA_OVERLAY`; otherwise uses `initialize_config_dir` + `compose(config_name=stem)` to load that YAML.
99- 
100-**Returns**:
101- 
102-An OmegaConf `DictConfig`. Sets `cfg.trainer.default_local_dir` to `/tmp/online_ppo_ckpt` by default, and calls `OmegaConf.resolve(cfg)` to interpolate variables before returning.
103- 
104-## class openjiuwen.agent_evolving.agent_rl.online.scheduler.ppo_executor.PPOTrainingExecutor
105- 
106-```python
107-class PPOTrainingExecutor(*, base_model_path: str, lora_repo: Optional[LoRARepository], notifier: Optional[InferenceNotifier], nproc_per_node: int, training_gpu_ids: str, ppo_config_path: Optional[str])
108-```
109- 
110-Owns the Ray/verl PPO runner lifecycle (lazy init, kill on close) and executes a single training batch.
111- 
112-**Parameters** (all keyword-only):
113- 
114-* **base_model_path**(str): Base model path.
115-* **lora_repo**(Optional[LoRARepository]): LoRA repository for publishing artifacts.
116-* **notifier**(Optional[InferenceNotifier]): vLLM hot-load notifier.
117-* **nproc_per_node**(int): Number of GPUs per node.
118-* **training_gpu_ids**(str): Comma-separated GPU IDs for training.
119-* **ppo_config_path**(Optional[str]): Custom PPO config path.
120- 
121-### async def aclose
122- 
123-```python
124-async def aclose() -> None
125-```
126- 
127-Closes the notifier (if present, swallowing exceptions) then calls `self.close()`. Used by the scheduler teardown.
128- 
129-### def close
130- 
131-```python
132-def close() -> None
133-```
134- 
135-If `_ppo_runner` is set, lazily imports `ray`, calls `ray.kill(self._ppo_runner, no_restart=True)` (swallowing exceptions), then resets `_ppo_runner`, `_ppo_initialized`. No-op if no runner.
136- 
137-### async def train_batch
138- 
139-```python
140-async def train_batch(*, user_id: str, samples: list[dict[str, Any]], training_count: int, tmp_root: str) -> Optional[str]
141-```
142- 
143-Executes one PPO training batch.
144- 
145-**Parameters** (all keyword-only):
146- 
147-* **user_id**(str): User identifier.
148-* **samples**(list[dict[str, Any]]): Training sample list.
149-* **training_count**(int): Training count (used for naming the run directory).
150-* **tmp_root**(str): Temp root directory.
151- 
152-**Returns**:
153- 
154-`Optional[str]`, the published LoRA path; `None` when `lora_repo` is not configured.
155- 
156-**Notes**:
157- 
158-- Creates `run_dir = Path(tmp_root)/f"run_{training_count}_{uuid.uuid4().hex[:8]}"`.
159-- Calls `_run_ppo_training_sync` via `asyncio.to_thread`.
160-- If a `published_lora_path` is returned and `notifier` is set, calls `notify_update(user_id, published_lora_path)` (failures are non-fatal).
161-- In `finally`, `shutil.rmtree(run_dir / "fsdp_ckpt", ignore_errors=True)`.
162- 
163-### def _init_ppo_trainer (private)
164- 
165-```python
166-def _init_ppo_trainer() -> None
167-```
168- 
169-Idempotent initialization. Lazily imports `ray`, `compose_online_ppo_config`, `OnlineTaskRunner`, `get_ppo_ray_runtime_env`; if Ray is not initialized, builds the runtime env (injecting `CUDA_VISIBLE_DEVICES`) and calls `ray.init(runtime_env=..., namespace="OnlineRL")`; composes the PPO config; creates the detached Ray actor `OnlineTaskRunner.options(name="online_ppo_runner", lifetime="detached").remote()`, calls `ray.get(self._ppo_runner.init_trainer.remote(config))`.
170- 
171-### def _run_ppo_training_sync (private)
172- 
173-```python
174-def _run_ppo_training_sync(*, user_id: str, samples: list[dict[str, Any]], run_dir: Path) -> Optional[str]
175-```
176- 
177-Synchronously executes training. Ensures trainer initialization; reads `pad_token_id` from `AutoTokenizer` (defaults to 0 on failure); reads `max_prompt_length`, `max_response_length`, `truncation` (default `"truncate"`), `filter_overlong_prompts` (default `False`) from `self._ppo_config.data`; constructs `VerlDataProtoConverter` to convert samples into `DataProto`; `ray.get(self._ppo_runner.train_on_batch.remote(data_proto))` trains; `ray.get(self._ppo_runner.export_lora.remote(str(run_dir), self.base_model_path))` exports LoRA; if `lora_repo` is set, publishes and returns the version path, otherwise returns `None`.
178- 
179-## Usage
180- 
181-- [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py): `start_online_training_scheduler` constructs `OnlineTrainingScheduler` and calls `.start()`.
182-- [runner.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/runner.py): Invokes the above function as step [3/5] of the launch sequence; calls `.stop()` in the shutdown path.
183-- [rl_optimizer.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/optimizer/rl_optimizer.py): `start_training()` constructs the scheduler; `train_on_batch(samples)` independently calls `compose_online_ppo_config`.
184-- [test_online_training_scheduler.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_online_training_scheduler.py): Disables polling with `redis_url=""`, injects fake store/trainer, directly calls private `_train_batch` to verify `mark_trained`/`mark_failed` behavior.
Mdocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/agent_rl.README.md+7-9
@@ -45,18 +45,16 @@
45 45 
46| CLASS / FUNCTION | DESCRIPTION |46| CLASS / FUNCTION | DESCRIPTION |
47|------------------|-------------|47|------------------|-------------|
48-| [GatewayConfig](./online/gateway.md) | Online-RL Gateway 运行时配置 dataclass。 |48+| `RLServiceConfig` | 独立 loopback RL Service 的静态配置。 |
49-| [build_app_from_config](./online/gateway.md) | 从 `GatewayConfig` 装配 FastAPI 应用的生产入口。 |49+| `build_rl_service_app` | 装配 Task、capture、trajectory 与 Training Run 路由的 RL Service 入口。 |
50-| [build_gateway_app](./online/gateway.md) | FastAPI 装配，注册路由（`/health`、`/v1/gateway/stats`、`/v1/gateway/upload/batch`、`/v1/chat/completions`、全量代理）。 |50+| `TaskRegistry` | Redis-backed Task、turn、capture 与 reward 状态所有者。 |
51-| [InferenceNotifier](./online/inference.md) | vLLM LoRA 热加载通知器。 |51+| `CapturePipeline` | 完整 OpenAI 请求/响应校验与带奖励轨迹发布。 |
52| [JudgeScorer](./online/judge.md) | LLM-as-a-Judge 高层异步评分客户端。 |52| [JudgeScorer](./online/judge.md) | LLM-as-a-Judge 高层异步评分客户端。 |
53-| [evaluate_judge_scores](./online/judge.md) | 核心评分入口，多投票平均并归一化到 `[-1, 1]`。 |53+| [evaluate_judge_scores](./online/judge.md) | 核心评分入口，多投票平均并归一化到 `[0, 1]`。 |
54-| [LauncherPaths](./online/launcher.md) | 在线 RL 循环编排的路径布局 dataclass。 |
55-| [run_online_rl_loop](./online/launcher.md) | 顶层编排入口，拉起并监管各服务进程。 |
56| [RLOnlineRail](./online/rail.md) | 钩入智能体生命周期的在线 RL 轨迹采集 Rail。 |54| [RLOnlineRail](./online/rail.md) | 钩入智能体生命周期的在线 RL 轨迹采集 Rail。 |
57| [TrajectoryUploader](./online/rail.md) | 异步上传 rail-v1 批次到 gateway 的上传器。 |55| [TrajectoryUploader](./online/rail.md) | 异步上传 rail-v1 批次到 gateway 的上传器。 |
58-| [OnlineTrainingScheduler](./online/scheduler.md) | 后台线程轮询 Redis 并触发 PPO 训练批次的调度器。 |56+| `TrainingRunner` | 显式固定 batch PPO、LoRA 激活、取消与恢复生命周期。 |
59-| [PPOTrainingExecutor](./online/scheduler.md) | Ray/verl PPO runner 生命周期与批次执行器。 |57+| `PPOTrainingExecutor` | 由显式 Training Run 调用的 Ray/verl PPO adapter。 |
60 58 
61**Functions**：59**Functions**：
62 60 
Ddocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/online/gateway.md+0-483
@@ -1,483 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.gateway
2- 
3-Online-RL Gateway：基于 FastAPI 的反向代理，位于 LLM 推理端点之前，记录每轮轨迹（含 LLM-as-Judge 评分），并暴露 rail-v1 批量上传端点。由在线 RL launcher 通过 uvicorn 工厂字符串启动。
4- 
5-子包结构：`upstream/`（传输与转发）→ `trajectory/`（持久化与摄入）→ `app/`（FastAPI 路由、装配、CLI/工厂）。唯一的生产入口是 uvicorn 工厂 `openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy:create_app`。
6- 
7-## class openjiuwen.agent_evolving.agent_rl.online.gateway.config.GatewayConfig
8- 
9-```python
10-@dataclass
11-class GatewayConfig(port: int, host: str = "127.0.0.1", llm_url: str = "http://127.0.0.1:18000", judge_url: str = "http://127.0.0.1:18001", model_id: str = "", judge_model: str = "", request_timeout: float = 120.0, llm_api_key: str = "", judge_api_key: str = "", gateway_api_key: str = "", record_dir: str = "records", log_level: str = "INFO", dump_token_ids: bool = False, lora_repo_root: str = "", redis_url: str = "", upstream_max_retries: int = 2, upstream_retry_backoff_sec: float = 0.2, upstream_retry_max_backoff_sec: float = 2.0, disable_gateway_trajectory_collection: bool = False, single_user_default: bool = True)
12-```
13- 
14-gateway 运行时配置 dataclass。
15- 
16-**字段**：
17- 
18-* **port**(int)：监听端口（必填）。
19-* **host**(str)：监听地址。默认值：`"127.0.0.1"`。
20-* **llm_url**(str)：上游 LLM 服务 URL。默认值：`"http://127.0.0.1:18000"`。
21-* **judge_url**(str)：judge LLM 服务 URL。默认值：`"http://127.0.0.1:18001"`。
22-* **model_id**(str)：模型 ID。默认值：`""`。
23-* **judge_model**(str)：judge 模型名。默认值：`""`。
24-* **request_timeout**(float)：请求超时（秒）。默认值：`120.0`。
25-* **llm_api_key**(str)：上游 LLM Bearer key。默认值：`""`。
26-* **judge_api_key**(str)：judge Bearer key。默认值：`""`。
27-* **gateway_api_key**(str)：gateway 自身鉴权 key。默认值：`""`。
28-* **record_dir**(str)：记录目录。默认值：`"records"`。
29-* **log_level**(str)：日志级别。默认值：`"INFO"`。
30-* **dump_token_ids**(bool)：是否在日志中输出 token ID。默认值：`False`。
31-* **lora_repo_root**(str)：LoRA 仓库根目录。默认值：`""`。
32-* **redis_url**(str)：Redis URL。默认值：`""`。
33-* **upstream_max_retries**(int)：上游最大重试次数。默认值：`2`。
34-* **upstream_retry_backoff_sec**(float)：上游重试退避基数（秒）。默认值：`0.2`。
35-* **upstream_retry_max_backoff_sec**(float)：上游重试最大退避（秒）。默认值：`2.0`。
36-* **disable_gateway_trajectory_collection**(bool)：禁用轨迹采集。默认值：`False`。
37-* **single_user_default**(bool)：单用户默认模式。默认值：`True`。
38- 
39-## 常量 NON_STANDARD_BODY_KEYS
40- 
41-```python
42-NON_STANDARD_BODY_KEYS: set[str] = {"session_id", "session_done", "turn_type", "memory_scope", "user_id", "workspace_id"}
43-```
44- 
45-转发前从 body 中剥离的非标准键。
46- 
47-## def openjiuwen.agent_evolving.agent_rl.online.gateway.common.utc_now_iso
48- 
49-```python
50-def utc_now_iso() -> str
51-```
52- 
53-返回当前 UTC 时间的 ISO-8601 字符串。
54- 
55-## def openjiuwen.agent_evolving.agent_rl.online.gateway.common.fit_list
56- 
57-```python
58-def fit_list(values: list[float], expected_len: int) -> list[float]
59-```
60- 
61-截断或用 `0.0` 填充 `values` 使其恰好有 `expected_len` 项；`expected_len <= 0` 时返回 `[]`。
62- 
63-## def openjiuwen.agent_evolving.agent_rl.online.gateway.message_utils.flatten_message_content
64- 
65-```python
66-def flatten_message_content(content: Any) -> str
67-```
68- 
69-将消息 content 归一化为字符串：`str` 原样返回；`list` 时以空格连接 `{"type": "text"}` 项的 `text`；`None` 返回 `""`；否则 `str(content)`。
70- 
71-## def openjiuwen.agent_evolving.agent_rl.online.gateway.message_utils.extract_last_user_instruction
72- 
73-```python
74-def extract_last_user_instruction(messages: list[dict]) -> str
75-```
76- 
77-逆序遍历 `messages`，返回最后一个 `role == "user"` 且文本非空消息的扁平化内容；无则返回 `""`。
78- 
79-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.upstream_client.RetryPolicy
80- 
81-```python
82-@dataclass(frozen=True)
83-class RetryPolicy(max_retries: int = 2, backoff_base_sec: float = 0.2, backoff_max_sec: float = 2.0)
84-```
85- 
86-上游重试策略冻结 dataclass。
87- 
88-### def backoff_for_attempt
89- 
90-```python
91-def backoff_for_attempt(attempt: int) -> float
92-```
93- 
94-指数退避 `backoff_base_sec * 2**(attempt-1)`，钳制到 `[0.0, backoff_max_sec]`；`attempt <= 0` 时返回 `0.0`。
95- 
96-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.upstream_client.UpstreamGatewayClient
97- 
98-```python
99-class UpstreamGatewayClient(Protocol)
100-```
101- 
102-上游传输客户端的结构化类型（`typing.Protocol`）接口。
103- 
104-### async def post_chat_completions
105- 
106-```python
107-async def post_chat_completions(*, json_body: dict[str, Any], headers: dict[str, str]) -> httpx.Response
108-```
109- 
110-POST `/v1/chat/completions`。
111- 
112-### async def request
113- 
114-```python
115-async def request(*, method: str, url: str, params: dict[str, Any], headers: dict[str, str], content: bytes) -> httpx.Response
116-```
117- 
118-发送任意请求。
119- 
120-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.upstream_client.HTTPXUpstreamGatewayClient
121- 
122-```python
123-class HTTPXUpstreamGatewayClient(*, http_client: httpx.AsyncClient, llm_url: str, retry_policy: RetryPolicy | None = None)
124-```
125- 
126-`UpstreamGatewayClient` 的 httpx 实现。
127- 
128-**参数**（均为关键字参数）：
129- 
130-* **http_client**(httpx.AsyncClient)：异步 HTTP 客户端。
131-* **llm_url**(str)：上游 LLM URL。
132-* **retry_policy**(RetryPolicy | None，可选)：重试策略，为 `None` 时使用默认。默认值：`None`。
133- 
134-### async def post_chat_completions
135- 
136-```python
137-async def post_chat_completions(*, json_body: dict[str, Any], headers: dict[str, str]) -> httpx.Response
138-```
139- 
140-POST 到 `{llm_url}/v1/chat/completions`，经 `_request_with_retry` 重试。
141- 
142-### async def request
143- 
144-```python
145-async def request(*, method: str, url: str, params: dict[str, Any], headers: dict[str, str], content: bytes) -> httpx.Response
146-```
147- 
148-通过 `http_client.request(...)` 发送任意请求，经 `_request_with_retry` 重试。
149- 
150-## class openjiuwen.agent_evolving.agent_rl.online.gateway.upstream.forwarder.Forwarder
151- 
152-```python
153-class Forwarder(*, upstream_client: UpstreamGatewayClient, model_id: str)
154-```
155- 
156-LLM 请求转发器。
157- 
158-**参数**（均为关键字参数）：
159- 
160-* **upstream_client**(UpstreamGatewayClient)：传输客户端。
161-* **model_id**(str)：默认模型 ID。
162- 
163-### async def forward
164- 
165-```python
166-async def forward(body: dict[str, Any], headers: dict[str, str]) -> dict[str, Any]
167-```
168- 
169-清理 body（移除 `NON_STANDARD_BODY_KEYS`、强制 `stream=False`、丢弃 `stream_options`、默认 `model`、设置 `logprobs=True`/`top_logprobs=1`），调用 `post_chat_completions`，`httpx.HTTPStatusError` 时抛 `HTTPException(502)`（detail 为响应文本前 500 字符），返回 `resp.json()`。
170- 
171-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.sample_recorder.SampleRecorder
172- 
173-```python
174-class SampleRecorder(*, sample_file: str, dump_token_ids: bool = False)
175-```
176- 
177-轻量样本计数与可选本地 JSONL 转储。
178- 
179-**参数**（均为关键字参数）：
180- 
181-* **sample_file**(str)：JSONL 文件路径。
182-* **dump_token_ids**(bool)：是否转储完整 token ID。默认值：`False`。
183- 
184-### async def record_sample
185- 
186-```python
187-async def record_sample(sample: dict[str, Any]) -> None
188-```
189- 
190-递增计数器；追加完整样本（`dump_token_ids=True`）或经 `_sample_for_log` 裁剪后的版本。
191- 
192-### async def snapshot_stats
193- 
194-```python
195-async def snapshot_stats() -> dict[str, int]
196-```
197- 
198-返回 `{"total_samples": self._total_samples}`。
199- 
200-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.pending_judge_store.PendingJudgeStore
201- 
202-```python
203-class PendingJudgeStore(*, redis: Any, ttl_sec: int = 24 * 3600)
204-```
205- 
206-Redis 后端的延迟 judge 待处理样本存储。`redis` 为 `None` 时抛 `ValueError`。
207- 
208-**参数**（均为关键字参数）：
209- 
210-* **redis**(Any)：Redis 客户端。
211-* **ttl_sec**(int)：每条样本 TTL（秒）。默认值：`86400`。
212- 
213-### async def put
214- 
215-```python
216-async def put(sample: dict[str, Any]) -> None
217-```
218- 
219-将样本 JSON 写入按样本键（带 TTL）并加入按会话的有序集合（按创建时间戳评分）。
220- 
221-### async def get_by_session
222- 
223-```python
224-async def get_by_session(session_id: str) -> list[dict[str, Any]]
225-```
226- 
227-通过 `ZRANGE` + `MGET` 返回某会话所有待处理样本（解码 bytes）。
228- 
229-### async def pop_one
230- 
231-```python
232-async def pop_one(session_id: str, trajectory_id: str, step_index: int) -> Optional[dict[str, Any]]
233-```
234- 
235-通过 pipeline 原子删除样本键并从会话有序集合移除，返回解码样本或 `None`。
236- 
237-### async def pop_earliest
238- 
239-```python
240-async def pop_earliest(session_id: str) -> Optional[dict[str, Any]]
241-```
242- 
243-通过 `get_by_session` + `pop_one` 返回最早的待处理样本。
244- 
245-### async def pop_all
246- 
247-```python
248-async def pop_all(session_id: str) -> list[dict[str, Any]]
249-```
250- 
251-弹出并返回某会话所有待处理样本。
252- 
253-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.judge_dispatcher.JudgeDispatcher
254- 
255-```python
256-class JudgeDispatcher(*, pending_store: Any, record_sample: Any, judge_scorer: Optional[Any] = None)
257-```
258- 
259-延迟 judge 分发器。
260- 
261-**参数**（均为关键字参数）：
262- 
263-* **pending_store**(Any)：待处理样本存储。
264-* **record_sample**(Any)：`record_sample` 协程。
265-* **judge_scorer**(Optional[Any]，可选)：`JudgeScorer` 实例。默认值：`None`。
266- 
267-### async def on_prev_feedback
268- 
269-```python
270-async def on_prev_feedback(session_id: str, prev_feedback: Optional[dict[str, Any]]) -> int
271-```
272- 
273-提取反馈文本；弹出该会话最早的待处理样本并终结（标签 `"prev_feedback"`）；记录；返回 1，否则 0。
274- 
275-### async def on_session_done
276- 
277-```python
278-async def on_session_done(session_id: str) -> int
279-```
280- 
281-弹出该会话所有待处理样本并逐个终结（最后一个标签 `"session_done"`，其余 `"session_flush"`）；记录；返回计数。
282- 
283-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.rail_ingest.RailBatchIngestor
284- 
285-```python
286-class RailBatchIngestor(*, pending_judge_store: Any, judge_dispatcher: Any, default_user_id: str = "")
287-```
288- 
289-rail-v1 批量上传摄入器。
290- 
291-**参数**（均为关键字参数）：
292- 
293-* **pending_judge_store**(Any)：待处理样本存储。
294-* **judge_dispatcher**(Any)：judge 分发器。
295-* **default_user_id**(str)：默认用户 ID。默认值：`""`。
296- 
297-### async def ingest_rail_batch
298- 
299-```python
300-async def ingest_rail_batch(payload: dict[str, Any]) -> dict[str, Any]
301-```
302- 
303-校验 `protocol_version == "rail-v1"`、必填 `session_id`/`trajectory_id`、`samples` 为列表；调用 `judge_dispatcher.on_prev_feedback(session_id, payload.get("prev_feedback"))`；逐个经 `_normalize_rail_sample` 归一化并放入待处理存储，统计 accepted/rejected（首个错误捕获）。全部拒绝时抛 `ValueError`。若 `payload.get("session_done")` 调用 `judge_dispatcher.on_session_done(session_id)`。返回 dict：`protocol_version`、`session_id`、`trajectory_id`、`accepted`、`rejected`、`judged`、`session_flushed`。
304- 
305-## class openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.persistence.GatewayTrajectoryRuntime
306- 
307-```python
308-class GatewayTrajectoryRuntime(config: Any, *, redis: Optional[Any] = None)
309-```
310- 
311-轨迹持久化与 rail 摄入装配；持有评分样本持久化。`redis` 为 `None` 时抛 `ValueError`。
312- 
313-**参数**：
314- 
315-* **config**(Any)：`GatewayConfig`（含 `record_dir`、`single_user_default`）。
316-* **redis**(Optional[Any]，可选)：Redis 客户端。默认值：`None`。
317- 
318-### @property def store_backend
319- 
320-```python
321-@property
322-def store_backend() -> str
323-```
324- 
325-返回 `type(self._trajectory_store).__name__`。
326- 
327-### @property def rail_ingestor
328- 
329-```python
330-@property
331-def rail_ingestor() -> RailBatchIngestor
332-```
333- 
334-返回摄入器；未初始化时抛 `RuntimeError`。
335- 
336-### def set_judge_scorer
337- 
338-```python
339-def set_judge_scorer(judge_scorer: Optional[Any]) -> None
340-```
341- 
342-重建 `JudgeDispatcher`（传入 `pending_store`、`record_sample`、`judge_scorer`）与 `RailBatchIngestor`（传入 `pending_judge_store`、`judge_dispatcher`、`default_user_id`）并存储。
343- 
344-### async def record_sample
345- 
346-```python
347-async def record_sample(sample: dict[str, Any]) -> None
348-```
349- 
350-归一化 `user_id`（默认 `_default_user_id`，缺失抛 `ValueError`）；保存到 Redis 轨迹存储并通过 `SampleRecorder` 记录。
351- 
352-### async def snapshot_stats
353- 
354-```python
355-async def snapshot_stats() -> dict[str, Any]
356-```
357- 
358-返回合并统计：`total_samples`、`trajectory_store_backend`、`trajectory_store_total`、`trajectory_store_pending`、`trajectory_store_training`、`trajectory_store_trained`、`trajectory_store_failed`。
359- 
360-## def openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.sample_payloads.build_sample
361- 
362-```python
363-def build_sample(*, user_id: str, session_id: str, turn_num: int, mode: str, io_mode: str, model: Any, messages: list[dict[str, Any]], tools: Any, assistant_message: dict[str, Any], usage: dict[str, Any], finish_reason: Optional[str], prompt_text: str, prompt_ids: list[int], response_text: str, response_ids: list[int], response_logprobs: list[float], tool_calls: list[dict[str, Any]], request_extras: Optional[dict[str, Any]] = None, sample_id: Optional[str] = None, created_at: Optional[str] = None, extra_fields: Optional[dict[str, Any]] = None) -> dict[str, Any]
364-```
365- 
366-构建归一化样本字典。包含 `sample_id`（默认 uuid4）、`created_at`（默认 `utc_now_iso()`）、`user_id`、`session_id`、`turn_num`、`mode`、`io_mode`、`model`、嵌套 `request`（messages、tools、**request_extras）、嵌套 `response`（message、usage、finish_reason）、嵌套 `trajectory`（`input_ids = prompt_ids + response_ids`、`attention_mask`、`response_mask`、`prompt_text`、`prompt_ids`、`response_text`、`response_ids`、`response_logprobs`、`tool_calls`）。合并 `extra_fields` 到顶层。
367- 
368-## def openjiuwen.agent_evolving.agent_rl.online.gateway.trajectory.sample_payloads.coerce_logprobs
369- 
370-```python
371-def coerce_logprobs(values: Any, expected_len: int) -> list[float]
372-```
373- 
374-将任意 logprob 值转为 float（跳过非数值），再 `fit_list(out, expected_len)` 到固定长度。
375- 
376-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.bootstrap.build_app_from_config
377- 
378-```python
379-def build_app_from_config(config: GatewayConfig, *, http_client: Any = None, redis_client: Any = None) -> FastAPI
380-```
381- 
382-从 `GatewayConfig` 装配 FastAPI 应用的生产入口。配置日志；创建/接受 Redis 异步客户端（需 `redis_url` 或注入 `redis_client`，否则抛 `ValueError`）；创建/接受 `httpx.AsyncClient`；构造 `HTTPXUpstreamGatewayClient`（带 `RetryPolicy`）；构造 `Forwarder`；构造 `GatewayTrajectoryRuntime(config, redis=redis_client)`；可选构造 `JudgeScorer`（当 `config.judge_url` 设置）并 `set_judge_scorer`；可选加载 `LoRARepository`；定义内部 `async def close_resources()` 关闭持有的 http/redis 客户端；返回 `build_gateway_app(config=..., forwarder=..., upstream_client=..., trajectory_runtime=..., close_resources=..., lora_repo=...)`。
383- 
384-**参数**：
385- 
386-* **config**(GatewayConfig)：配置。
387-* **http_client**(Any，可选)：注入的 HTTP 客户端。默认值：`None`。
388-* **redis_client**(Any，可选)：注入的 Redis 客户端。默认值：`None`。
389- 
390-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.server.build_gateway_app
391- 
392-```python
393-def build_gateway_app(*, config: Any, forwarder: Forwarder, upstream_client: UpstreamGatewayClient, trajectory_runtime: GatewayTrajectoryRuntime, close_resources: Callable[[], Awaitable[None]], lora_repo: Any = None) -> FastAPI
394-```
395- 
396-创建 `FastAPI(title="Online-RL Gateway", lifespan=...)`，lifespan 中调用 `close_resources()`。注册路由：
397- 
398-- `GET /health` → `{"status": "ok"}`
399-- `GET /v1/gateway/stats` → 鉴权；返回 `_snapshot_stats(...)` 与当前请求计数（`asyncio.Lock` 保护）。
400-- `POST /v1/gateway/upload/batch` → 鉴权；调用 `trajectory_runtime.rail_ingestor.ingest_rail_batch(payload)`，返回 `{"ok": True, "result": result}`；`ValueError` → `HTTPException(400)`。
401-- `POST /v1/chat/completions` → 鉴权；递增计数；解析 JSON body；解析 `user_id`；注入最新 LoRA；弹出 `stream`；调用 `_forward_chat_completions`；流式时返回 `StreamingResponse(stream_chat_response(...))`，否则 `JSONResponse`。
402-- `GET/POST/PUT/PATCH/DELETE/OPTIONS/HEAD /{path:path}`（全量代理）→ 鉴权；转发至 `upstream_client.request(...)` 目标 `{config.llm_url}/{path}`，剥离逐跳头，返回 `Response`。
403- 
404-**参数**（均为关键字参数）：
405- 
406-* **config**(Any)：`GatewayConfig`。
407-* **forwarder**(Forwarder)：转发器。
408-* **upstream_client**(UpstreamGatewayClient)：上游客户端。
409-* **trajectory_runtime**(GatewayTrajectoryRuntime)：轨迹运行时。
410-* **close_resources**(Callable[[], Awaitable[None]])：关闭资源协程。
411-* **lora_repo**(Any，可选)：LoRA 仓库。默认值：`None`。
412- 
413-## async def openjiuwen.agent_evolving.agent_rl.online.gateway.app.http_helpers.ensure_gateway_auth
414- 
415-```python
416-async def ensure_gateway_auth(gateway_api_key: str, authorization: Optional[str]) -> None
417-```
418- 
419-`gateway_api_key` 为空时空操作；否则要求 `Bearer ` token 匹配，缺失抛 `HTTPException(401)`，不匹配抛 `HTTPException(403)`。
420- 
421-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.http_helpers.build_upstream_headers
422- 
423-```python
424-def build_upstream_headers(request: Request, *, llm_api_key: str) -> dict[str, str]
425-```
426- 
427-复制入站请求头，丢弃 `host`/`content-length`/`connection` 与所有 `x-forwarded-*`；`llm_api_key` 设置时注入 `Authorization: Bearer {llm_api_key}`。
428- 
429-## async def openjiuwen.agent_evolving.agent_rl.online.gateway.app.http_helpers.stream_chat_response
430- 
431-```python
432-async def stream_chat_response(response_json: dict[str, Any], *, model_id: str)
433-```
434- 
435-异步生成器（产出 SSE 字符串），将非流式 chat 响应包装为合成 SSE 流：首个 chunk（含 delta role/content/tool_calls/reasoning_content、token_ids、logprobs、prompt_token_ids）、最终 chunk（含 `finish_reason` 与 `usage`），随后 `data: [DONE]`。
436- 
437-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.request_context.resolve_trace_id
438- 
439-```python
440-def resolve_trace_id(request: Request) -> str
441-```
442- 
443-返回 `x-request-id` 头，无则合成 `uuid.uuid4().hex[:8]`。
444- 
445-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.request_context.require_messages
446- 
447-```python
448-def require_messages(body: dict[str, Any]) -> list[dict[str, Any]]
449-```
450- 
451-校验 `body["messages"]` 为非空列表；否则抛 `HTTPException(400)`。
452- 
453-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.request_context.require_user_id
454- 
455-```python
456-def require_user_id(request: Request, config: Any) -> str
457-```
458- 
459-读取 `x-user-id` 头；为空且 `config.single_user_default` 为真时回退默认 ID；仍为空抛 `HTTPException(400)`。
460- 
461-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy.create_app
462- 
463-```python
464-def create_app()
465-```
466- 
467-uvicorn 工厂入口（`uvicorn ...gateway.app.proxy:create_app --factory`）。经 `_build_config_from_env()` 构建配置后返回 `build_app_from_config(config)`。
468- 
469-## def openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy.main
470- 
471-```python
472-def main() -> None
473-```
474- 
475-CLI 入口。解析参数（`--host`、`--port` 必填，`--llm-url`、`--judge-url`、`--model-id`、`--judge-model`、`--record-dir`、`--lora-repo-root`、`--log-level`），构造 `GatewayConfig`，调用 `build_app_from_config`，`uvicorn.run(...)`。
476- 
477-## 被使用情况
478- 
479-- [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py)：以字符串 `DEFAULT_GATEWAY_APP_FACTORY = 'openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy:create_app'` 引用 gateway，通过 `uvicorn ... --factory` 子进程启动。这是 gateway 唯一的生产入口。
480-- [judge_scorer.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/judge/judge_scorer.py)：提供 `JudgeScorer`，由 `app/bootstrap.py` 构造，`JudgeDispatcher._finalize_sample` 调用 `score(...)`。
481-- [redis_trajectory_store.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/storage/redis_trajectory_store.py)：提供 `RedisTrajectoryStore`，由 `trajectory/persistence.py` 构造，`save_sample(...)` 与 `stats()` 被 `GatewayTrajectoryRuntime` 使用。
482-- [lora_repo.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/storage/lora_repo.py)：提供 `LoRARepository`，可选由 `app/bootstrap.py` 构造，`get_latest(user_id)` 由 `app/server.py` 的 `_inject_latest_lora` 调用。
483-- 测试：[test_forwarder.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_forwarder.py)、[test_processor_components.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_processor_components.py)、[test_upstream_client.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_upstream_client.py)、[test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py)、[test_online_gateway_e2e.py](file:///Users/dongdong/Desktop/project/agent-core/tests/system_tests/agent_evolving/agent_rl/online/test_online_gateway_e2e.py)。
Ddocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/online/inference.md+0-90
@@ -1,90 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.inference
2- 
3-vLLM 推理服务的 LoRA 热加载通知器。通过调用 vLLM 原生的 `/v1/load_lora_adapter` 与 `/v1/unload_lora_adapter` 端点，在不重启服务的前提下为指定用户热加载（或卸载）LoRA 适配器。在线 RL 训练完成并发布 LoRA 后，由调度器调用此类通知 vLLM 应用新权重。
4- 
5-## class openjiuwen.agent_evolving.agent_rl.online.inference.notifier.InferenceNotifier
6- 
7-```python
8-class InferenceNotifier(vllm_base_url: str, timeout: float = 120.0, http_client: Optional[httpx.AsyncClient] = None)
9-```
10- 
11-异步 HTTP 客户端，负责通知 vLLM 热加载/卸载 LoRA 适配器。
12- 
13-**参数**：
14- 
15-* **vllm_base_url**(str)：vLLM 服务的基础 URL（末尾 `/` 会被去除），例如 `http://vllm.local`。
16-* **timeout**(float)：单次请求超时（秒）。默认值：`120.0`。
17-* **http_client**(Optional[httpx.AsyncClient]，可选)：外部注入的异步 HTTP 客户端。为 `None` 时，通知器会创建并持有自己的 `httpx.AsyncClient`。默认值：`None`。
18- 
19-**说明**：
20- 
21-- 当传入外部 `http_client` 时，通知器不持有该客户端，`close()` 不会关闭它；自行创建的客户端则由通知器持有并在关闭时释放。
22- 
23-### async def close
24- 
25-```python
26-async def close() -> None
27-```
28- 
29-关闭底层 HTTP 客户端。
30- 
31-**说明**：
32- 
33-- 仅当通知器自行创建（持有）HTTP 客户端时才会执行关闭；若客户端由外部注入，则该方法为空操作。可在异步上下文中安全调用。
34- 
35-### async def notify_update
36- 
37-```python
38-async def notify_update(user_id: str, lora_path: str) -> None
39-```
40- 
41-通知 vLLM 为指定用户热加载 LoRA 适配器。
42- 
43-**参数**：
44- 
45-* **user_id**(str)：用户标识符，同时作为 vLLM 的 `lora_name`。加载后，请求中指定该 `lora_name` 即可自动应用新权重。
46-* **lora_path**(str)：LoRA 权重目录的绝对路径。
47- 
48-**说明**：
49- 
50-- 向 `{vllm_base_url}/v1/load_lora_adapter` 发送 HTTP POST，JSON body 为：
51- 
52-```json
53-{
54- "lora_name": "<user_id>",
55- "lora_path": "<lora_path>",
56- "load_inplace": true
57-}
58-```
59- 
60-- 使用 `self.timeout` 作为请求超时。
61-- 当 HTTP 状态码 `>= 400` 时，抛出 `RuntimeError`，消息形如 `vLLM load_lora_adapter failed: status=<code>, body=<body 前 400 字符>`。
62-- 成功时记录 INFO 日志：`LoRA hot-loaded for user %s: %s`。
63- 
64-### async def unload
65- 
66-```python
67-async def unload(user_id: str) -> None
68-```
69- 
70-卸载指定用户的 LoRA 适配器（可用于清理非活跃用户）。
71- 
72-**参数**：
73- 
74-* **user_id**(str)：要卸载的 `lora_name`（用户标识符）。
75- 
76-**说明**：
77- 
78-- 向 `{vllm_base_url}/v1/unload_lora_adapter` 发送 HTTP POST，JSON body 为 `{"lora_name": "<user_id>"}`。
79-- 通过 `resp.raise_for_status()` 抛出 HTTP 错误（与 `notify_update` 的自定义 `RuntimeError` 不同）。
80-- 成功时记录 INFO 日志：`LoRA unloaded for user %s`。
81- 
82-## 被使用情况
83- 
84-`InferenceNotifier` 在以下位置被构造或调用：
85- 
86-- [ppo_executor.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/scheduler/ppo_executor.py)：`PPOTrainingExecutor` 构造时接收 `Optional[InferenceNotifier]`；`aclose()` 中调用 `close()`；LoRA 发布成功后调用 `notify_update(user_id, published_lora_path)`（失败视为非致命警告）。
87-- [online_training_scheduler.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/scheduler/online_training_scheduler.py)：构造 `PPOTrainingExecutor` 时透传 `notifier`。
88-- [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py)：`start_online_training_scheduler` 中构造 `InferenceNotifier(runtime.inference_url)` 并传入调度器。
89-- [rl_optimizer.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/optimizer/rl_optimizer.py)：`setup_inference(vllm_url)` 存储 URL，构建时构造 `InferenceNotifier` 并传入 `OnlineTrainingScheduler`。
90-- [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py)：测试注入伪造 `httpx.AsyncClient` 验证 `notify_update` 的请求路径与 body，并验证外部客户端不被 `close()` 关闭。
Mdocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/online/judge.md+8-10
@@ -1,6 +1,6 @@
1# openjiuwen.agent_evolving.agent_rl.online.judge1# openjiuwen.agent_evolving.agent_rl.online.judge
2 2 
3-LLM-as-a-Judge 奖励评分器。沿四个维度（任务完成度、响应质量、工具使用、连贯性，各 0-10 分）对智能体单轮响应评分，归一化为 `[-1, 1]` 奖励，支持多投票平均与重试逻辑。是 gateway 轨迹流水线（延迟 judge 流）消费的奖励来源。3+LLM-as-a-Judge 奖励评分器。沿四个维度（任务完成度、响应质量、工具使用、连贯性，各 0-10 分）对智能体单轮响应评分，归一化为 `[0, 1]` 奖励，支持多投票平均与重试逻辑。是 RL Service capture 流水线（延迟 judge 流）消费的奖励来源。
4 4 
5模块内部分三层：`scoring.py`（纯 prompt/解析/归一化，无 I/O）→ `evaluator.py`（异步 HTTP 投票引擎）→ `judge_scorer.py`（高层客户端适配器）；`judge_server.py` 为可选独立 FastAPI 服务。5模块内部分三层：`scoring.py`（纯 prompt/解析/归一化，无 I/O）→ `evaluator.py`（异步 HTTP 投票引擎）→ `judge_scorer.py`（高层客户端适配器）；`judge_server.py` 为可选独立 FastAPI 服务。
6 6 
@@ -53,7 +53,7 @@ def parse_judge_scores(content: str, *, raise_on_error: bool = True) -> Optional
53def normalize_overall_score(overall: float) -> float53def normalize_overall_score(overall: float) -> float
54```54```
55 55 
56-将原始 0-10 分映射到 `[-1, 1]`，公式为 `(overall - 5.0) / 5.0`。56+将原始 0-10 分映射到 `[0, 1]`，公式为 `overall / 10.0`。
57 57 
58**参数**：58**参数**：
59 59 
@@ -61,7 +61,7 @@ def normalize_overall_score(overall: float) -> float
61 61 
62**返回**：62**返回**：
63 63 
64-`float`，归一化后的 `[-1, 1]` 奖励。64+`float`，归一化后的 `[0, 1]` 奖励。
65 65 
66## class openjiuwen.agent_evolving.agent_rl.online.judge.evaluator.JudgeEvaluatorConfig66## class openjiuwen.agent_evolving.agent_rl.online.judge.evaluator.JudgeEvaluatorConfig
67 67 
@@ -108,7 +108,7 @@ async def evaluate_judge_scores(*, client: httpx.AsyncClient, config: JudgeEvalu
108 108 
109```python109```python
110{110{
111- "score": float, # 归一化后 [-1, 1]111+ "score": float, # 归一化后 [0, 1]
112 "overall_raw": float, # 平均原始 0-10 分112 "overall_raw": float, # 平均原始 0-10 分
113 "votes": list[float], # 各投票的 overall 值113 "votes": list[float], # 各投票的 overall 值
114 "details": dict | list, # num_votes==1 时为单个投票字典，否则为列表114 "details": dict | list, # num_votes==1 时为单个投票字典，否则为列表
@@ -249,9 +249,7 @@ def main() -> None
249 249 
250## 被使用情况250## 被使用情况
251 251 
252-- [bootstrap.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/app/bootstrap.py)：当 `config.judge_url` 设置时构造 `JudgeScorer` 并注入 `trajectory_runtime.set_judge_scorer(judge_scorer)`。这是 `JudgeScorer` 唯一的生产构造点。252+- `service.py` 根据 RL Service 配置构造 `JudgeScorer`，并注入 `CapturePipeline`。
253-- [persistence.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/trajectory/persistence.py)：`set_judge_scorer` 将其传入新的 `JudgeDispatcher`。253+- `capture_pipeline.py` 在原子发布完整 delayed-feedback turn 时调用 `JudgeScorer.score(...)`。
254-- [judge_dispatcher.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/trajectory/judge_dispatcher.py)：`_finalize_sample` 中调用 `await self._judge_scorer.score(...)`（异常回退 `{"score": 0.0, ...}`）。这是 `JudgeScorer.score` 唯一的生产调用点。254+- `test_gateway_support.py` 覆盖分数解析和 `finish_reason=="length"` 重试路径。
255-- [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py)：测试 `JudgeScorer._parse_scores` 与 `finish_reason=="length"` 重试路径。255+- `test_capture_pipeline.py` 覆盖 scorer 接口、原子发布以及 Judge 失败处理。
256-- [test_processor_components.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/gateway/test_processor_components.py)：定义 duck-typed `_FakeJudgeScorer` 验证 `.score(...)` 接口。
257-- 注意：`judge_server.py` 的 `create_app`/`main`/`JudgeConfig`/`ScoreRequest`/`ScoreResponse` **未被任何模块导入**。launcher 以原始 vLLM 进程启动 judge，而非此 FastAPI 服务；该模块为可选独立服务（`python -m judge.judge_server ...`）。
Ddocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/online/launcher.md+0-369
@@ -1,369 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.launcher
2- 
3-JiuwenClaw 在线 RL 循环的编排运行时（interact → trajectory collect → PPO train → LoRA hot-load）。负责解析 CLI 参数、合并配置、拉起各服务进程（vLLM 推理、judge、gateway、训练调度器、JiuwenClaw 应用与 web），并通过信号处理与健康检查实现优雅关闭。
4- 
5-模块分层清晰：`cli.py` 解析参数，`loader.py` 合并配置，`services.py` 拉起进程，`workspace.py` 写入环境文件，`runner.py` 编排循环，`__init__.py` 重导出顶层入口。
6- 
7-## class openjiuwen.agent_evolving.agent_rl.online.launcher.runner.LauncherPaths
8- 
9-```python
10-@dataclass(frozen=True)
11-class LauncherPaths(agent_core_root: Path, jiuwenclaw_repo: Path, workspace_root: Path, workspace_env: Path, script_dir: Path)
12-```
13- 
14-描述运行器所需的磁盘布局的冻结 dataclass。
15- 
16-**字段**：
17- 
18-* **agent_core_root**(Path)：agent-core 仓库根目录。
19-* **jiuwenclaw_repo**(Path)：JiuwenClaw 仓库根目录。
20-* **workspace_root**(Path)：工作区根目录。
21-* **workspace_env**(Path)：工作区 `.env` 文件路径。
22-* **script_dir**(Path)：脚本目录（日志目录在其下）。
23- 
24-## def openjiuwen.agent_evolving.agent_rl.online.launcher.runner.run_online_rl_loop
25- 
26-```python
27-def run_online_rl_loop(*, cfg: OnlineRLConfig, cfg_path: Path, paths: LauncherPaths) -> None
28-```
29- 
30-顶层编排入口。安装 SIGINT/SIGTERM 处理器（抛出 `_ShutdownRequested`），执行启动序列：
31- 
32-1. 创建 `logs/` 目录。
33-2. `resolve_launch_runtime(cfg, script_dir=paths.script_dir)`。
34-3. 预检所需端口。
35-4. 启动推理 vLLM（`enable_runtime_lora=True`），除非 `runtime.skip_vllm`。
36-5. 启动 judge vLLM（`enable_runtime_lora=False`），除非 `runtime.skip_judge`。
37-6. 健康检查 vLLM 与 judge。
38-7. 启动 gateway，健康检查 gateway。
39-8. 通过 `start_online_training_scheduler` 启动训练调度器。
40-9. 若 `cfg.jiuwen.enabled`：`ensure_workspace` 后 `start_jiuwenclaw`；否则跳过。
41-10. `print_launch_summary`。
42-11. 监管循环：每 30 秒轮询各子进程 `poll()`，任一退出则停止全部并返回。
43-12. `finally` 中 `_shutdown()` 依次停止调度器、终止 web→claw→gateway→judge→vllm（幂等）。
44- 
45-**参数**（均为关键字参数）：
46- 
47-* **cfg**(OnlineRLConfig)：运行配置。
48-* **cfg_path**(Path)：配置文件路径。
49-* **paths**(LauncherPaths)：路径布局。
50- 
51-## def openjiuwen.agent_evolving.agent_rl.online.launcher.cli.build_arg_parser
52- 
53-```python
54-def build_arg_parser() -> argparse.ArgumentParser
55-```
56- 
57-构造 CLI 参数解析器，描述为 `'JiuwenClaw online RL loop: interact -> trajectory collect -> PPO train -> LoRA hot-load'`。
58- 
59-**返回**：
60- 
61-`argparse.ArgumentParser`。主要参数（dest / 类型 / 默认值）：
62- 
63-| Flag | Dest | 类型 | 默认值 |
64-|------|------|------|--------|
65-| `--config` | `config` | str | None |
66-| `--model-path` | `model_path` | str | None |
67-| `--model-name` | `model_name` | str | None |
68-| `--vllm-gpu` | `vllm_gpu` | str | None |
69-| `--vllm-tp` | `vllm_tp` | int | None |
70-| `--vllm-port` | `vllm_port` | int | None |
71-| `--judge-model-path` | `judge_model_path` | str | None |
72-| `--judge-model-name` | `judge_model_name` | str | None |
73-| `--judge-gpu` | `judge_gpu` | str | None |
74-| `--judge-tp` | `judge_tp` | int | None |
75-| `--judge-port` | `judge_port` | int | None |
76-| `--gateway-port` | `gateway_port` | int | None |
77-| `--redis-url` | `redis_url` | str | None |
78-| `--threshold` | `threshold` | int | None |
79-| `--scan-interval` | `scan_interval` | int | None |
80-| `--train-gpu` | `train_gpu` | str | None |
81-| `--ppo-config` | `ppo_config` | str | None |
82-| `--trajectory-batch-size` | `trajectory_batch_size` | int | None |
83-| `--lora-repo` | `lora_repo` | str | None |
84-| `--jiuwen-agent-server-port` | `jiuwen_agent_server_port` | int | None |
85-| `--demo` | `demo` | store_true | None |
86-| `--inference-url` | `inference_url` | str | None |
87-| `--judge-url` | `judge_url` | str | None |
88-| `--skip-jiuwen`/`--skip_jiuwen` | `skip_jiuwen` | store_true | False |
89-| `--jiuwen-ws-port` | `jiuwen_ws_port` | int | None |
90-| `--jiuwen-web-host` | `jiuwen_web_host` | str | None |
91-| `--jiuwen-web-port` | `jiuwen_web_port` | int | None |
92- 
93-## def openjiuwen.agent_evolving.agent_rl.online.launcher.cli.build_cli_overrides
94- 
95-```python
96-def build_cli_overrides(args: argparse.Namespace) -> dict[str, object]
97-```
98- 
99-将解析后的 `Namespace` 转换为嵌套覆盖字典。使用硬编码 `cli_mappings` 表将 CLI 属性名映射为点分配置路径（如 `model_path` → `inference.model_path`），跳过 `None` 值。`--skip-jiuwen` 为真时设置 `jiuwen.enabled` 为 `False`。
100- 
101-**参数**：
102- 
103-* **args**(argparse.Namespace)：解析后的参数。
104- 
105-**返回**：
106- 
107-`dict[str, object]`，嵌套覆盖字典。
108- 
109-## def openjiuwen.agent_evolving.agent_rl.online.launcher.loader.load_runtime_config
110- 
111-```python
112-def load_runtime_config(*, config_path: str | None, cli_overrides: dict[str, object]) -> tuple[OnlineRLConfig, Path]
113-```
114- 
115-三层配置合并（OmegaConf）：内置 `BUILTIN_ONLINE_RL_CONFIG` + 可选 YAML（`config_path` 缺失时回退到内置 `online_config.py`）+ CLI 覆盖。返回 Pydantic 校验后的 `OnlineRLConfig` 与解析后的路径。
116- 
117-**参数**（均为关键字参数）：
118- 
119-* **config_path**(str | None)：用户 YAML 路径，为 `None` 时使用内置默认。
120-* **cli_overrides**(dict[str, object])：CLI 覆盖字典。
121- 
122-**返回**：
123- 
124-`tuple[OnlineRLConfig, Path]`，配置对象与解析后的配置文件路径。
125- 
126-## def openjiuwen.agent_evolving.agent_rl.online.launcher.loader.resolve_builtin_online_config_path
127- 
128-```python
129-def resolve_builtin_online_config_path() -> Path
130-```
131- 
132-通过模块 `__file__` 定位 `online_config.py` 的磁盘路径。
133- 
134-**返回**：
135- 
136-`Path`，内置配置文件路径。无法定位时抛 `RuntimeError`。
137- 
138-## 常量 DEFAULT_CONFIG_FILENAME
139- 
140-```python
141-DEFAULT_CONFIG_FILENAME = "online_config.py (built-in)"
142-```
143- 
144-用于 CLI 帮助文本，指示配置来源。
145- 
146-## class openjiuwen.agent_evolving.agent_rl.online.launcher.services.LaunchRuntime
147- 
148-```python
149-@dataclass(frozen=True)
150-class LaunchRuntime(inference_url: str, judge_url: str, gateway_base_url: str, gateway_api_url: str, lora_repo: str, skip_vllm: bool, skip_judge: bool, reuse_inference_for_judge: bool, judge_label: str, ports_to_check: tuple[tuple[str, str, int], ...])
151-```
152- 
153-持有运行器消费的解析后 URL/标志的冻结 dataclass。`ports_to_check` 每项为 `(name, host, port)`。
154- 
155-**字段**：
156- 
157-* **inference_url**(str)：推理服务 URL。
158-* **judge_url**(str)：judge 服务 URL。
159-* **gateway_base_url**(str)：gateway 基础 URL。
160-* **gateway_api_url**(str)：gateway API URL。
161-* **lora_repo**(str)：LoRA 仓库根目录。
162-* **skip_vllm**(bool)：是否跳过 vLLM 启动。
163-* **skip_judge**(bool)：是否跳过 judge 启动。
164-* **reuse_inference_for_judge**(bool)：是否复用推理服务作为 judge。
165-* **judge_label**(str)：judge 标签。
166-* **ports_to_check**(tuple[tuple[str, str, int], ...])：待预检端口列表。
167- 
168-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.resolve_launch_runtime
169- 
170-```python
171-def resolve_launch_runtime(cfg: OnlineRLConfig, *, script_dir: Path) -> LaunchRuntime
172-```
173- 
174-从配置解析服务 URL、跳过标志、端口检查列表与 LoRA 仓库位置。
175- 
176-**参数**：
177- 
178-* **cfg**(OnlineRLConfig)：运行配置。
179-* **script_dir**(Path)：脚本目录（用于解析 LoRA 仓库相对路径）。
180- 
181-**返回**：
182- 
183-`LaunchRuntime`。
184- 
185-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.url_host
186- 
187-```python
188-def url_host(host: str) -> str
189-```
190- 
191-将通配绑定主机（`'0.0.0.0'`、`'::'`）规范化为 `'127.0.0.1'`，用于客户端 URL 构造。
192- 
193-**参数**：
194- 
195-* **host**(str)：绑定主机。
196- 
197-**返回**：
198- 
199-`str`，规范化后的主机。
200- 
201-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.spawn_process
202- 
203-```python
204-def spawn_process(cmd: list[str], *, env: dict[str, str] | None = None, cwd: str | None = None, log_path: Path | None = None) -> subprocess.Popen
205-```
206- 
207-通用子进程启动器。提供 `log_path` 时将 stdout+stderr 追加到该文件（自动创建父目录），否则继承父进程 stdio。
208- 
209-**参数**：
210- 
211-* **cmd**(list[str])：命令列表。
212-* **env**(dict[str, str] | None，可选)：环境变量。默认值：`None`。
213-* **cwd**(str | None，可选)：工作目录。默认值：`None`。
214-* **log_path**(Path | None，可选)：日志文件路径。默认值：`None`。
215- 
216-**返回**：
217- 
218-`subprocess.Popen`。
219- 
220-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_vllm_service
221- 
222-```python
223-def start_vllm_service(service_cfg: VLLMServiceConfig, *, step_label: str, service_name: str, enable_runtime_lora: bool, log_path: Path | None = None) -> subprocess.Popen
224-```
225- 
226-启动 vLLM OpenAI API server（`python -m vllm.entrypoints.openai.api_server`），参数取自 `service_cfg`。设置 `CUDA_VISIBLE_DEVICES`；当 `enable_runtime_lora=True` 时设置 `VLLM_ALLOW_RUNTIME_LORA_UPDATING=1`。
227- 
228-**参数**（均为关键字参数，除 `service_cfg`）：
229- 
230-* **service_cfg**(VLLMServiceConfig)：服务配置。
231-* **step_label**(str)：步骤标签（日志用）。
232-* **service_name**(str)：服务名（日志用）。
233-* **enable_runtime_lora**(bool)：是否启用运行时 LoRA 更新。
234-* **log_path**(Path | None，可选)：日志路径。默认值：`None`。
235- 
236-**返回**：
237- 
238-`subprocess.Popen`。
239- 
240-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_gateway
241- 
242-```python
243-def start_gateway(*, inference_url: str, judge_url: str, judge_model: str, model_id: str, model_path: str, lora_repo_root: str, gateway_cfg: GatewayServiceConfig, agent_core_root: Path, log_path: Path | None = None) -> subprocess.Popen
244-```
245- 
246-通过 `python -m uvicorn <DEFAULT_GATEWAY_APP_FACTORY> --factory ...` 启动 agent-core gateway。设置 `LLM_URL`、`JUDGE_URL`、`JUDGE_MODEL`、`MODEL_ID`、`MODEL_PATH`、`GATEWAY_HOST/PORT`、`RECORD_DIR`、`REDIS_URL`、可选 `LORA_REPO_ROOT`、可选 `DISABLE_GATEWAY_TRAJECTORY_COLLECTION` 等环境变量，以 `cwd=agent_core_root` 运行。
247- 
248-**参数**（均为关键字参数）：
249- 
250-* **inference_url**(str)：推理 URL。
251-* **judge_url**(str)：judge URL。
252-* **judge_model**(str)：judge 模型名。
253-* **model_id**(str)：模型 ID。
254-* **model_path**(str)：模型路径。
255-* **lora_repo_root**(str)：LoRA 仓库根目录。
256-* **gateway_cfg**(GatewayServiceConfig)：gateway 服务配置。
257-* **agent_core_root**(Path)：agent-core 根目录。
258-* **log_path**(Path | None，可选)：日志路径。默认值：`None`。
259- 
260-**返回**：
261- 
262-`subprocess.Popen`。
263- 
264-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_online_training_scheduler
265- 
266-```python
267-def start_online_training_scheduler(*, cfg: OnlineRLConfig, runtime: LaunchRuntime)
268-```
269- 
270-惰性导入 `InferenceNotifier`、`OnlineTrainingScheduler`、`LoRARepository`，构造并启动调度器（轮询 `RedisTrajectoryStore`，当待处理轨迹达 `threshold` 时触发 PPO LoRA 训练）。返回调度器实例，调用 `scheduler.start()` 后返回。
271- 
272-**参数**（均为关键字参数）：
273- 
274-* **cfg**(OnlineRLConfig)：运行配置。
275-* **runtime**(LaunchRuntime)：解析后的运行时信息。
276- 
277-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.start_jiuwenclaw
278- 
279-```python
280-def start_jiuwenclaw(*, jiuwenclaw_repo: Path, workspace_root: Path, trajectory_gateway_url: str, model_path: str, trajectory_mode: str, trajectory_batch_size: int, app_host: str, ws_port: int, web_host: str, web_port: int) -> tuple[subprocess.Popen, subprocess.Popen | None]
281-```
282- 
283-启动 JiuwenClaw 应用（`python -m jiuwenclaw.app`），可选启动 web 前端（当存在 `web/dist` 目录时运行 `python -m jiuwenclaw.app_web`）。从环境解析 `RL_ONLINE_TENANT_ID`（回退 `WEB_USER_ID` 或 `'local-web-user'`），注入 `WEB_USER_ID` 与 JSON 编码的 `CUSTOM_HEADERS`（`{'x-user-id': ...}`）。使用 `build_trajectory_env_updates` 构造轨迹环境变量。
284- 
285-**参数**（均为关键字参数）：
286- 
287-* **jiuwenclaw_repo**(Path)：JiuwenClaw 仓库根目录。
288-* **workspace_root**(Path)：工作区根目录。
289-* **trajectory_gateway_url**(str)：轨迹 gateway URL。
290-* **model_path**(str)：模型路径。
291-* **trajectory_mode**(str)：轨迹模式。
292-* **trajectory_batch_size**(int)：轨迹批大小。
293-* **app_host**(str)：应用监听地址。
294-* **ws_port**(int)：WebSocket 端口。
295-* **web_host**(str)：web 前端监听地址。
296-* **web_port**(int)：web 前端端口。
297- 
298-**返回**：
299- 
300-`tuple[subprocess.Popen, subprocess.Popen | None]`，应用进程与可选 web 进程。
301- 
302-## def openjiuwen.agent_evolving.agent_rl.online.launcher.services.print_launch_summary
303- 
304-```python
305-def print_launch_summary(*, cfg: OnlineRLConfig, cfg_path: Path, runtime: LaunchRuntime, web_started: bool) -> None
306-```
307- 
308-输出格式化的启动摘要（配置路径、web/WS URL、vLLM 推理/judge URL、gateway、Redis、轨迹模式/日志、LoRA 仓库、训练阈值、批大小、扫描间隔、训练 GPU、使用提示）。web 前端行根据 `cfg.jiuwen.enabled` 与 `web_started` 条件性包含。
309- 
310-**参数**（均为关键字参数）：
311- 
312-* **cfg**(OnlineRLConfig)：运行配置。
313-* **cfg_path**(Path)：配置文件路径。
314-* **runtime**(LaunchRuntime)：运行时信息。
315-* **web_started**(bool)：web 是否已启动。
316- 
317-## 常量
318- 
319-```python
320-DEFAULT_GATEWAY_APP_FACTORY = 'openjiuwen.agent_evolving.agent_rl.online.gateway.app.proxy:create_app'
321-EXISTING_SERVICE_HEALTH_TIMEOUT = 30.0
322-```
323- 
324-- `DEFAULT_GATEWAY_APP_FACTORY`：uvicorn 应用工厂目标字符串。
325-- `EXISTING_SERVICE_HEALTH_TIMEOUT`：复用外部管理服务时的健康检查超时（秒）。
326- 
327-## def openjiuwen.agent_evolving.agent_rl.online.launcher.workspace.build_trajectory_env_updates
328- 
329-```python
330-def build_trajectory_env_updates(*, gateway_url: str, model_path: str, trajectory_batch_size: int, trajectory_mode: str, trajectory_tenant_id: str | None = None) -> dict[str, str]
331-```
332- 
333-返回 JiuwenClaw Rail（在线 RL 轨迹上传）所需的环境变量字典。
334- 
335-**参数**（均为关键字参数）：
336- 
337-* **gateway_url**(str)：gateway URL。
338-* **model_path**(str)：模型路径。
339-* **trajectory_batch_size**(int)：轨迹批大小。
340-* **trajectory_mode**(str)：轨迹模式。
341-* **trajectory_tenant_id**(str | None，可选)：租户 ID。默认值：`None`。
342- 
343-**返回**：
344- 
345-`dict[str, str]`，包含 `USE_RL_ONLINE_RAIL='1'`、`ENABLE_TRAJECTORY_COLLECTION='false'`、`TRAJECTORY_GATEWAY_URL`、`TRAJECTORY_TOKENIZER_PATH`、`TRAJECTORY_BATCH_SIZE`、`TRAJECTORY_MODE`，以及可选的 `RL_ONLINE_TENANT_ID`。
346- 
347-## def openjiuwen.agent_evolving.agent_rl.online.launcher.workspace.ensure_workspace
348- 
349-```python
350-def ensure_workspace(*, config_env: Path, gateway_url: str, model_name: str, model_path: str, trajectory_mode: str, trajectory_gateway_url: str | None = None, trajectory_batch_size: int = 8) -> None
351-```
352- 
353-确保 JiuwenClaw `.env` 文件指向 gateway。文件不存在时惰性导入并调用 `jiuwenclaw.utils.prepare_workspace(overwrite=False, preferred_language='zh')`；随后合并（保留既有键）一组值：`API_BASE`、`API_KEY='EMPTY'`、`MODEL_NAME`、`MODEL_PROVIDER='OpenAI'`、`WEB_USER_ID`、`CUSTOM_HEADERS`（JSON）、`EMBED_*`、`BROWSER_RUNTIME_MCP_ENABLED='0'`、`EVOLUTION_AUTO_SCAN='false'`，以及 `build_trajectory_env_updates` 的结果。写回完整文件。
354- 
355-**参数**（均为关键字参数）：
356- 
357-* **config_env**(Path)：`.env` 文件路径。
358-* **gateway_url**(str)：gateway URL。
359-* **model_name**(str)：模型名。
360-* **model_path**(str)：模型路径。
361-* **trajectory_mode**(str)：轨迹模式。
362-* **trajectory_gateway_url**(str | None，可选)：轨迹 gateway URL，为 `None` 时使用 `gateway_url`。默认值：`None`。
363-* **trajectory_batch_size**(int)：轨迹批大小。默认值：`8`。
364- 
365-## 被使用情况
366- 
367-- [run_online_rl.py](file:///Users/dongdong/Desktop/project/agent-core/examples/jiuwenrl_online/run_online_rl.py)：用户可见的启动脚本。导入 `build_arg_parser`、`build_cli_overrides`、`load_runtime_config`、`LauncherPaths`、`run_online_rl_loop`，构造 `LauncherPaths` 后调用主入口。这是唯一的生产外部消费者。
368-- [test_launcher_runner.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_launcher_runner.py)：测试 `run_online_rl_loop` 信号关闭、`print_launch_summary`、`ensure_workspace`、`start_jiuwenclaw`。
369-- [README.md](file:///Users/dongdong/Desktop/project/agent-core/examples/jiuwenrl_online/README.md)：引用 `print_launch_summary` 输出。
Mdocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/online/rail.md+4-6
@@ -304,9 +304,7 @@ def build_rl_online_rail_from_env() -> Optional[RLOnlineRail]
304 304 
305## 被使用情况305## 被使用情况
306 306 
307-- [\_\_init\_\_.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/__init__.py)：通过 `__getattr__` 懒导出 `RLOnlineRail`（规范外部导入路径为 `openjiuwen.agent_evolving.agent_rl.RLOnlineRail`），并列入 `__all__`。307+- `openjiuwen.agent_evolving.agent_rl.__init__` 懒导出 `RLOnlineRail`，作为规范外部导入路径。
308-- [workspace.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/workspace.py) 与 [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py)：为生成的 JiuwenClaw 进程设置 `USE_RL_ONLINE_RAIL=1`、`TRAJECTORY_GATEWAY_URL`、`RL_ONLINE_TENANT_ID` 等环境变量（不直接导入 rail 类）。308+- AIGW 持有公开 `POST /v1/gateway/upload/batch` 入口，并将批次代理到运行中的 RL Service。
309-- [server.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/app/server.py)：注册 `POST /v1/gateway/upload/batch` 端点，接收上传；由 [rail_ingest.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/gateway/trajectory/rail_ingest.py) 中的 `RailBatchIngestor` 消费。309+- Agent 生命周期属于外部应用；应用按需直接配置 `RLOnlineRail`，RL Service 不拉起 Agent。
310-- [test_rl_online_rail.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_rl_online_rail.py)：导入并构造 `RLOnlineRail`。310+- `test_rl_online_rail.py` 覆盖 `RLOnlineRail`、`TrajectoryUploader` 与 `OnlineTrajectoryConverter`。
311-- [test_online_gateway_e2e.py](file:///Users/dongdong/Desktop/project/agent-core/tests/system_tests/agent_evolving/agent_rl/online/test_online_gateway_e2e.py)：通过 `importlib` 动态导入 `RLOnlineRail` 与 `TrajectoryUploader`。
312-- [test_gateway_support.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_gateway_support.py)：导入 `OnlineTrajectoryConverter` 并测试 `convert` / `to_dict`。
Ddocs/zh/2.开发指南/API文档/openjiuwen.agent_evolving/agent_rl/online/scheduler.md+0-184
@@ -1,184 +0,0 @@
1-# openjiuwen.agent_evolving.agent_rl.online.scheduler
2- 
3-在线 RL 训练的轮询调度与 PPO 批次执行组件。后台线程轮询 Redis 轨迹存储中累积的样本，当某用户的样本数达到阈值时触发一次 PPO LoRA 训练批次：转换样本 → 调用 Ray/verl 训练 → 导出 LoRA → 发布到仓库 → 通知 vLLM 热加载。
4- 
5-## class openjiuwen.agent_evolving.agent_rl.online.scheduler.online_training_scheduler.OnlineTrainingScheduler
6- 
7-```python
8-class OnlineTrainingScheduler(*, redis_url: str = "redis://127.0.0.1:6379/0", poll_interval: float = 30.0, min_samples_for_training: int = 32, base_model_path: str = "", lora_repo: Optional[LoRARepository] = None, notifier: Optional[InferenceNotifier] = None, nproc_per_node: int = 1, training_gpu_ids: str = "", tmp_root: str = "/tmp/agent_rl_online", ppo_config_path: Optional[str] = None)
9-```
10- 
11-后台线程调度器。轮询 `RedisTrajectoryStore`，对样本数超过阈值的用户启动一个 asyncio 任务运行 PPO 训练批次，同一时刻最多保留一个在途训练任务。
12- 
13-**参数**（均为关键字参数）：
14- 
15-* **redis_url**(str)：Redis 连接 URL。默认值：`"redis://127.0.0.1:6379/0"`。为空字符串时禁用调度器（测试中使用）。
16-* **poll_interval**(float)：轮询间隔（秒）。默认值：`30.0`。
17-* **min_samples_for_training**(int)：触发训练的最小样本数阈值。默认值：`32`。
18-* **base_model_path**(str)：基座模型路径。默认值：`""`。
19-* **lora_repo**(Optional[LoRARepository]，可选)：LoRA 仓库实例，用于发布训练产物。默认值：`None`。
20-* **notifier**(Optional[InferenceNotifier]，可选)：vLLM 热加载通知器。默认值：`None`。
21-* **nproc_per_node**(int)：单节点 GPU 数。默认值：`1`。
22-* **training_gpu_ids**(str)：训练用 GPU ID 列表（逗号分隔）。默认值：`""`。
23-* **tmp_root**(str)：训练临时根目录。默认值：`"/tmp/agent_rl_online"`。
24-* **ppo_config_path**(Optional[str]，可选)：自定义 Hydra PPO YAML 路径。默认值：`None`。
25- 
26-**说明**：
27- 
28-- 构造时内部创建一个 `PPOTrainingExecutor` 并传入 `base_model_path`、`lora_repo`、`notifier`、`nproc_per_node`、`training_gpu_ids`、`ppo_config_path`。
29- 
30-### def start
31- 
32-```python
33-def start() -> None
34-```
35- 
36-启动守护轮询线程（线程名 `OnlineTrainScheduler`，目标为 `_poll_loop`）。若已在运行则记录警告并无操作。
37- 
38-### def stop
39- 
40-```python
41-def stop() -> None
42-```
43- 
44-发送停止信号、等待线程（15 秒超时）后调用 `self._trainer.close()`。若线程仍存活则记录警告。幂等。
45- 
46-### async def _poll_loop (私有)
47- 
48-```python
49-async def _poll_loop() -> None
50-```
51- 
52-后台线程入口。创建专用 asyncio 事件循环，惰性导入 `redis.asyncio.from_url`，构建 `RedisTrajectoryStore`，运行 `_poll_main`；`finally` 中关闭 trainer、Redis 客户端与循环。当 `redis_url` 为空时直接返回。
53- 
54-### async def _poll_main (私有)
55- 
56-```python
57-async def _poll_main() -> None
58-```
59- 
60-主循环：未停止时依次执行 `_reap_training_task()` → `_poll_once()` → `sleep(poll_interval)`；循环结束后 `await _reap_training_task(wait=True)` 排空在途任务。每轮异常被捕获并记录。
61- 
62-### async def _poll_once (私有)
63- 
64-```python
65-async def _poll_once() -> None
66-```
67- 
68-当存储为空或已有在途训练任务时为空操作。否则调用 `get_users_above_threshold(min_samples_for_training)`，对首个可获取样本的用户调用 `fetch_and_mark_training(user_id, min_samples_for_training)`，创建 `asyncio.create_task(self._train_batch(...))`，每个周期最多启动一个训练任务。
69- 
70-### async def _reap_training_task (私有)
71- 
72-```python
73-async def _reap_training_task(*, wait: bool = False) -> None
74-```
75- 
76-收割在途训练任务。无任务时直接返回；除非 `wait=True`，任务未完成时提前返回。否则 await 该任务，异常被记录，`finally` 中清理 `_active_training_task` 与 `_active_training_user`。
77- 
78-### async def _train_batch (私有)
79- 
80-```python
81-async def _train_batch(*, user_id: str, samples: list[dict[str, Any]], sample_ids: list[str]) -> None
82-```
83- 
84-执行单个训练批次。调用 `self._trainer.train_batch(user_id=..., samples=..., training_count=self._training_count, tmp_root=self.tmp_root)`；成功时 `mark_trained(sample_ids)`，异常时 `mark_failed(sample_ids)`。存储为空时为空操作。
85- 
86-## def openjiuwen.agent_evolving.agent_rl.online.scheduler.ppo_config.compose_online_ppo_config
87- 
88-```python
89-def compose_online_ppo_config(*, model_path: str, n_gpus_per_node: int = 2, config_path: Optional[str] = None)
90-```
91- 
92-组合在线 PPO 训练的 Hydra/OmegaConf 配置。
93- 
94-**参数**（均为关键字参数）：
95- 
96-* **model_path**(str)：基座模型路径，写入 `cfg.actor_rollout_ref.model.path`。
97-* **n_gpus_per_node**(int)：单节点 GPU 数，写入 `cfg.trainer.n_gpus_per_node`。默认值：`2`。
98-* **config_path**(Optional[str]，可选)：用户自定义 Hydra YAML 路径。为 `None` 时加载 verl 内置 `ppo_trainer` 配置并合并 `ONLINE_PPO_VERL_HYDRA_OVERLAY`；否则使用 `initialize_config_dir` + `compose(config_name=stem)` 加载该 YAML。
99- 
100-**返回**：
101- 
102-OmegaConf `DictConfig`。设置 `cfg.trainer.default_local_dir` 默认为 `/tmp/online_ppo_ckpt`，并调用 `OmegaConf.resolve(cfg)` 解析变量后返回。
103- 
104-## class openjiuwen.agent_evolving.agent_rl.online.scheduler.ppo_executor.PPOTrainingExecutor
105- 
106-```python
107-class PPOTrainingExecutor(*, base_model_path: str, lora_repo: Optional[LoRARepository], notifier: Optional[InferenceNotifier], nproc_per_node: int, training_gpu_ids: str, ppo_config_path: Optional[str])
108-```
109- 
110-持有 Ray/verl PPO runner 生命周期（惰性初始化、关闭时 kill）并执行单个训练批次。
111- 
112-**参数**（均为关键字参数）：
113- 
114-* **base_model_path**(str)：基座模型路径。
115-* **lora_repo**(Optional[LoRARepository])：LoRA 仓库，用于发布产物。
116-* **notifier**(Optional[InferenceNotifier])：vLLM 热加载通知器。
117-* **nproc_per_node**(int)：单节点 GPU 数。
118-* **training_gpu_ids**(str)：训练用 GPU ID 列表。
119-* **ppo_config_path**(Optional[str])：自定义 PPO 配置路径。
120- 
121-### async def aclose
122- 
123-```python
124-async def aclose() -> None
125-```
126- 
127-关闭通知器（若存在，吞掉异常）后调用 `self.close()`。用于调度器拆卸。
128- 
129-### def close
130- 
131-```python
132-def close() -> None
133-```
134- 
135-若 `_ppo_runner` 已设置，惰性导入 `ray`，调用 `ray.kill(self._ppo_runner, no_restart=True)`（吞掉异常），随后重置 `_ppo_runner`、`_ppo_initialized`。无 runner 时为空操作。
136- 
137-### async def train_batch
138- 
139-```python
140-async def train_batch(*, user_id: str, samples: list[dict[str, Any]], training_count: int, tmp_root: str) -> Optional[str]
141-```
142- 
143-执行一次 PPO 训练批次。
144- 
145-**参数**（均为关键字参数）：
146- 
147-* **user_id**(str)：用户标识符。
148-* **samples**(list[dict[str, Any]])：训练样本列表。
149-* **training_count**(int)：训练计数（用于命名运行目录）。
150-* **tmp_root**(str)：临时根目录。
151- 
152-**返回**：
153- 
154-`Optional[str]`，发布的 LoRA 路径；未配置 `lora_repo` 时返回 `None`。
155- 
156-**说明**：
157- 
158-- 创建 `run_dir = Path(tmp_root)/f"run_{training_count}_{uuid.uuid4().hex[:8]}"`。
159-- 通过 `asyncio.to_thread` 调用 `_run_ppo_training_sync`。
160-- 若返回 `published_lora_path` 且 `notifier` 已设置，调用 `notify_update(user_id, published_lora_path)`（失败非致命）。
161-- `finally` 中 `shutil.rmtree(run_dir / "fsdp_ckpt", ignore_errors=True)`。
162- 
163-### def _init_ppo_trainer (私有)
164- 
165-```python
166-def _init_ppo_trainer() -> None
167-```
168- 
169-幂等初始化。惰性导入 `ray`、`compose_online_ppo_config`、`OnlineTaskRunner`、`get_ppo_ray_runtime_env`；若 Ray 未初始化则构建 runtime env（注入 `CUDA_VISIBLE_DEVICES`）并 `ray.init(runtime_env=..., namespace="OnlineRL")`；组合 PPO 配置；创建 detached Ray actor `OnlineTaskRunner.options(name="online_ppo_runner", lifetime="detached").remote()`，调用 `ray.get(self._ppo_runner.init_trainer.remote(config))`。
170- 
171-### def _run_ppo_training_sync (私有)
172- 
173-```python
174-def _run_ppo_training_sync(*, user_id: str, samples: list[dict[str, Any]], run_dir: Path) -> Optional[str]
175-```
176- 
177-同步执行训练。确保 trainer 初始化；从 `AutoTokenizer` 读取 `pad_token_id`（失败默认 0）；从 `self._ppo_config.data` 读取 `max_prompt_length`、`max_response_length`、`truncation`（默认 `"truncate"`）、`filter_overlong_prompts`（默认 `False`）；构造 `VerlDataProtoConverter` 转换样本为 `DataProto`；`ray.get(self._ppo_runner.train_on_batch.remote(data_proto))` 训练；`ray.get(self._ppo_runner.export_lora.remote(str(run_dir), self.base_model_path))` 导出 LoRA；若 `lora_repo` 已设置则发布并返回版本路径，否则返回 `None`。
178- 
179-## 被使用情况
180- 
181-- [services.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/services.py)：`start_online_training_scheduler` 构造 `OnlineTrainingScheduler` 并 `.start()`。
182-- [runner.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/online/launcher/runner.py)：启动序列第 [3/5] 步调用上述函数，关闭路径调用 `.stop()`。
183-- [rl_optimizer.py](file:///Users/dongdong/Desktop/project/agent-core/openjiuwen/agent_evolving/agent_rl/optimizer/rl_optimizer.py)：`start_training()` 构造调度器；`train_on_batch(samples)` 独立调用 `compose_online_ppo_config`。
184-- [test_online_training_scheduler.py](file:///Users/dongdong/Desktop/project/agent-core/tests/unit_tests/agent_evolving/agent_rl/online/test_online_training_scheduler.py)：以 `redis_url=""` 禁用轮询，注入伪造 store/trainer，直接调用私有 `_train_batch` 验证 `mark_trained`/`mark_failed` 行为。
Mopenjiuwen/agent_evolving/agent_rl/config/__init__.py+2-0
Mopenjiuwen/agent_evolving/agent_rl/online/__init__.py+29-0
Mopenjiuwen/agent_evolving/agent_rl/online/gateway/__init__.py+2-4
Dopenjiuwen/agent_evolving/agent_rl/online/gateway/app/__init__.py+0-4
Dopenjiuwen/agent_evolving/agent_rl/online/gateway/collector/__init__.py+0-26
Dopenjiuwen/agent_evolving/agent_rl/online/gateway/upstream/__init__.py+0-14
Dopenjiuwen/agent_evolving/agent_rl/online/launcher/__init__.py+0-8
Mopenjiuwen/agent_evolving/agent_rl/online/scheduler/__init__.py+3-19
Mopenjiuwen/agent_evolving/agent_rl/storage/__init__.py+27-26
Mpyproject.toml+2-0
Muv.lock+2-0