| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(infra): bind forked workers to reserved ports (#1657) * fix(infra): bind forked workers to reserved ports Keep process and port ownership coupled across allocation, fork, health checks, and cleanup so failed starts and kills remain retryable. Migrate schedulers and service controllers to the owner-bound Guard protocol and roll back partial startup failures. * fix(infra): make Guard owner lifecycle atomic Serialize owner-bound allocation, fork, release, and kill state transitions so concurrent requests cannot orphan children or release ports from a newer process generation. Normalize worker indices across lifecycle endpoints and roll back partially acquired node port locks on allocation failure. | 1 个月前 | |
fix: stabilize Qwen Flash Next multimodal RL (#1774) * fix: stabilize Qwen Flash Next multimodal RL Preserve original image and generated token identities across proxy rollouts and keep long-context training inputs aligned with the actor. Retain valid Arena rollout groups through transient failures and preserve frozen parameters across checkpoint and AWEX ownership transitions. Remove diagnostic-only replay, stable top-k and dump helpers from the production recipe. Keep the latest main dependency versions. Signed-off-by: chucai.dzq <chucai.dzq@antgroup.com> * fix(examples): prevent multimodal preprocessing requests from hanging * fix(api): preserve EOS-only completions for trajectory export * fix(examples): require loaded state for Flash Next recovery * test: add HTTP status to RTensor delete response mock * fix(examples): fetch receipts for abbreviated Arena failures * fix(infra): propagate checkpoint failures through engine RPC * test: align Flash Next fixtures with recovery and image patches * fix(engine): align multimodal preprocessing and address review * refactor(engine): consolidate pinned SGLang source patches * fix: make SGLang patches repeatable and remove stale entry * fix(engine): apply Qwen SGLang patches at server startup Select the Qwen Flash Next profile from raw model config, independent of workflow and model directory name. Prepare the inference interpreter before SGLang imports, then exec the existing v2, AWEX or native entry. Move QSA and image preprocessing patches into shared engine modules. Validate the whole source set before writing, serialize concurrent starts, and atomically replace only changed files. Preserve non-HF loader paths and allow already patched read-only runtimes to start. Remove the recipe's manual patch commands and cover model selection, route preservation, unknown sources, concurrent starts and repetition. * docs(examples): remove Qwen Flash Next README * feat(examples): tag Arena tasks with experiment and rollout step Group Arena tasks by the configured experiment and the rollout weight version captured before asynchronous task setup. Key changes: - Forward experiment and metadata through both Arena launch APIs - Wire experiment names from both Arena training entrypoints - Cover zero steps, version changes, and scalar-client forwarding Refs: #1774 * fix(examples): match Arena launch metadata schema Arena's live OpenAPI requires string values in task metadata. Encode the rollout weight version as a string and reject invalid metadata before submission in both launch APIs. Refs: #1774 * refactor: remove unused replay and graybox prompt workarounds The diagnostic replay implementation has been removed, so its RPC alias-preservation exception has no remaining caller. Keep the two supported rollout boundaries in the whitelist. Graybox output instructions belong in the Harness or task prompt. Drop the unused training-side preprocessor and its tests; no recipe enables it. Refs: #1774 * refactor(experimental): simplify episode reward eligibility checks Extract full-chain validation so the reward assignment stays readable. Keep rejection of ambiguous and explicitly scored chains atomic, with coverage for invalid IDs, templates, and existing reward annotations. --------- Signed-off-by: chucai.dzq <chucai.dzq@antgroup.com> Co-authored-by: root <root@gpulingjun033180164218.et15> | 5 天前 | |
feat: introduce Daytona cloud sandbox backend (#1231) * feat(sandbox): introduce Daytona cloud sandbox backend * fix(sandbox): prevent Daytona sandbox lifecycle leaks | 5 个月前 | |
feat(infra): add distributed data loading service (#1120) * feat(infra): add distributed data loading service Introduce a controller/router/gateway/worker data service so single-controller training can offload dataset access to remote workers while preserving existing trainer workflows. Key changes: - add RDataset-based remote dataset registration and fetch path - integrate DataController lifecycle into RL/SFT/RW trainers - add guard reuse, data-service APIs, and full infra test coverage * chore: revert stats_logger to make trackio test happy | 6 个月前 | |
feat(launcher): add best-effort post-exit hook (#1459) Run a configurable post-exit shell command after local, Ray, and Slurm launchers stop jobs on timeout, interrupt, or failure paths. The hook receives LOG_DIR, has a 600-second timeout, and logs failures without interrupting shutdown or recovery. Ensure the hook still runs if launcher shutdown raises. Add the config field, unit tests, and regenerated CLI docs. | 3 个月前 | |
fix(infra): preserve LD_PRELOAD in local launchers (#1578) Avoid wrapping local targets with stdbuf when the effective target environment already defines LD_PRELOAD. GNU stdbuf appends libstdbuf to the value, which breaks TMS CDLL loading during offload. Preserve env=None versus explicit env={} semantics while retaining line buffering for targets without LD_PRELOAD. Refs: #1570 Signed-off-by: yaoyaoshiguonan <yaoyaoguonan@outlook.com> | 2 个月前 | |
fix(infra): reap detached Slurm worker descendants (#1771) Co-authored-by: root <root@gpulingjun033180164218.et15> | 9 天前 | |
fix(infra): reap detached Slurm worker descendants (#1771) Co-authored-by: root <root@gpulingjun033180164218.et15> | 9 天前 | |
fix(infra): fail fast when a local inference server dies during launch (#1545) * fix(infra): fail fast when a local inference server dies during launch _wait_for_server already receives the server process handle but never inspects it: when the server crashes at startup (port conflict, scheduler init error, OOM kill), the launcher keeps polling the health endpoint until setup_timeout expires and then raises a bare TimeoutError. Poll the process each iteration and raise immediately with the exit code and a pointer to the server traceback instead, which also makes the scheduler-level launch retries practical since a failed attempt now surfaces in seconds rather than after the full timeout. Log a heartbeat every 60s while waiting and include the address and timeout in the TimeoutError message. * test(infra): cover fail-fast on dead inference server process Requested in review: mock process.poll() returning an exit code and assert that _wait_for_server raises RuntimeError carrying the returncode instead of waiting for the health-check timeout. * fix(infra): shut the server down on any launch failure The fail-fast path raises RuntimeError, which escaped the TimeoutError handler in launch_server and skipped _shutdown_one_server. The skip happens to be harmless today because the address is not registered yet and the dead process needs no kill, but the two failure modes should not diverge. Catch both and report the underlying reason instead of assuming a timeout. | 2 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 5 天前 | ||
| 5 个月前 | ||
| 6 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 9 天前 | ||
| 9 天前 | ||
| 2 个月前 |