| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(examples): retune GSM8K GRPO learning rate for FP32 master weights (#1634) Signed-off-by: Bo Yang <yb550079@antgroup.com> | 22 天前 | |
fix: apply_chat_template compatibility with transformers>=5.0 (#1280) * fix: apply_chat_template compatibility with transformers>=5.0 * chore: revert async client in inference controller * chore: fix async client | 4 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
refactor(api): migrate allocation_mode to per-engine backend fields (#1044) * refactor(api): migrate allocation_mode to per-engine backend fields Replace the centralized `allocation_mode` string with explicit `backend` fields on `TrainEngineConfig` and `InferenceEngineConfig`. Each engine now owns its own backend+parallelism spec (e.g. `fsdp:d4`, `sglang:d4t2`), eliminating implicit auto-backend selection and the shared `AllocationMode` object. Key changes: - Add `backend` field to TrainEngineConfig and InferenceEngineConfig - Add `ModelAllocation.from_str()` for single-component parsing - Remove `AllocationMode` public export (replaced by `ModelAllocation`) - Rename internal `AllocationMode` to `_AllocationMode` for SPMD launcher backward compatibility with FutureWarning - Remove auto-backend selection — explicit backend prefix is now required - Controllers (`TrainController`, `RolloutController`) parse `backend` directly instead of receiving `alloc_mode` from trainers - `WeightUpdateMeta.alloc_mode` replaced by `gen_allocation` (single `ModelAllocation`) - Add `RWTrainer` and `ArchonRWEngine` for reward model training - Remove `get_model_update_meta()` helper (logic moved to trainers) - Update all YAML configs, examples, docs (EN+ZH), and tests BREAKING CHANGE: `AllocationMode` is removed from public API. Users must migrate to per-engine `backend` fields. SPMD launchers emit deprecation warnings. * chore(ci): fix backend specifier for vlm sft test * fix: fix bare dims for actor backends * chore(docs): fix reminder for bare allocation dims | 5 个月前 | |
fix(examples): retune GSM8K GRPO learning rate for FP32 master weights (#1634) Signed-off-by: Bo Yang <yb550079@antgroup.com> | 22 天前 | |
feat(colocate): support AWEX colocated actor-rollout training (#1500) * feat(colocate): support AWEX colocated actor-rollout training Add shared-GPU colocated training: the Megatron actor and the SGLang rollout engine time-share the same devices, coordinated through AWEX weight transfer. - AWEX weight-sync writer with tag-based offload/onload and an SGLang engine plugin implementing the colocate reader protocol. - Colocation scheduling support in the Slurm scheduler and controllers (pause/resume generation around the train phase, engine-level offload orchestration, recover handling). - Megatron model registration and fast HF checkpoint loading for the Bailing hybrid MoE family. Both sides key the CUDA IPC handoff on physical GPU ids, mapping through the device mask. Deriving them differently only agrees when the mask starts at zero, so a process pinned to any other GPU would await a key its peer never publishes. Pausing for a weight update keeps SGLang's default mode, which aborts in-flight requests and returns their partial output so the client resumes them by extending the prompt. A second in-place pause raises the scheduler's paused flag so the colocate loop services awex work. Splitting it this way leaves the scheduler fully idle, which SGLang requires before releasing memory, and other backends keep their single-request pause. Supports SGLang 0.5.9 and 0.5.10, whose removed decode-stat hooks are now optional. Verified on multi-node colocated RL runs: training statistics align with the separated-GPU baseline, and single-batch replays produce bitwise-identical training stats. A single-node example test covers the colocate path end to end. * fix(colocate): gate colocation-only steps on the v1 AWEX colocate setup weight_update_mode alone does not identify the colocated v1 run. Controller v2 selects AWEX from use_lora and never reads that field, so a v2 separation run may legitimately carry weight_update_mode="awex" and would then take the v1 colocation handover: its offload was disabled, a stray meta server started, awex_colocate_mode was forced onto its SGLang servers, and both checkpoint branches were skipped so it wrote nothing at all. Gate on _is_v1_awex_colocate, which also requires controller v1 and an actor-rollout colocation strategy. The weight-update meta dispatch keeps its comparison: it sits in an elif chain that v2 short-circuits earlier. Behaviour is unchanged for every configuration that exists today; the three colocated example configs still select the path and the separation config still does not. * refactor(colocate): configure the allocator per role instead of at import AWEX_ACTOR_ALLOC_CONF existed because the colocate examples point rollout at ${actor.scheduling_spec}, so both roles shared one env_vars mapping while the actor wants expandable_segments and SGLang's memory saver cannot tolerate it. Applying it required rewriting PYTORCH_CUDA_ALLOC_CONF from the first lines of the top-level package, because the `from .infra` chain initializes CUDA and freezes the allocator config; that put argv sniffing and an environment mutation into every `import areal`, and the same block was duplicated in rpc_server. Give each colocated role its own scheduling_spec env_vars instead: the env then reaches the process through `srun --env`, before it starts, so no import-time hook is needed. Drop the mechanism and both copies. The mirror of it in the SGLang plugin never worked. It ran from the __main__ block, long after the module-level `from areal.utils import ...` had already frozen the allocator config, so it rewrote the variable while allocations stayed expandable. Replace it with an assertion that runs before any areal import and fails loudly, since a silently self-disabled memory saver surfaces much later as a colocate OOM or an invalid CUDA IPC target. * perf(mcore): build the gloo mirror group only when the engine can offload resolve_broadcast_target reads cpu_model_parallel_group only after an offloaded engine has handed the accelerator to rollout and device collectives are unusable. Creating it unconditionally cost one gloo new_group per data-parallel group at startup on every run, including separation runs that never offload. Gate it on the engine's offload config. Nothing changes on the consumer side: the attribute already defaults to None and resolve_broadcast_target falls back to the device group in that case. * fix(colocate): do not require a colocation strategy to select v1 AWEX The gate added earlier also demanded an actor-rollout colocation scheduling strategy. AWEX runs opt in through weight_update_mode and leave actor and rollout on the default separation strategy, so the gate went false for every such run: the controller never started the AWEX meta server nor passed its address to the rollout, each training worker then started a server of its own, and the rollout registered against a different one. The run waited on 'infer_conf' until the timeout with no error. Keep the controller-version check, which is what stops a v2 separation run from taking this path, and drop the strategy check. The truth table in the test was asserting the broken behaviour, so it is corrected alongside. * docs(examples): add the colocated AWEX GRPO config Mirrors the two-GPU setup the example test exercises: a Megatron actor and an SGLang rollout time-sharing both GPUs, with weights handed over through AWEX. The TMS switches live in the actor's own scheduling_spec env_vars rather than a global toggle: SGLang opens its own memory-saver regions for the colocated rollout, and an auto-opened region on the training side would nest inside them. * refactor: drop the unrelated rpc_server import reformat The parenthesised import left behind by removing the AWEX allocator environment knob carries no semantic change, so rpc_server.py no longer needs to appear in this change set at all. * refactor(recover): fold the colocate helpers into RecoverHandler Both helpers are only reachable from RecoverHandler.load(), so keeping them at module level widened the public surface of areal.utils.recover for no caller. They become private static methods next to the existing _ensure_recover_supported/_normalize_recover_engines pair. | 1 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
fix: preserve vllm lora aliases and local model paths (#1601) Keep prior versioned LoRA names registered in the vLLM server so in-flight rollout requests do not break after adapter refreshes. Also fixed lora example yamls as per some latest syntax. Key changes: - Retain old runtime LoRA aliases in areal_vllm_server.py - Add a regression test for alias retention | 26 天前 | |
fix: preserve vllm lora aliases and local model paths (#1601) Keep prior versioned LoRA names registered in the vLLM server so in-flight rollout requests do not break after adapter refreshes. Also fixed lora example yamls as per some latest syntax. Key changes: - Retain old runtime LoRA aliases in areal_vllm_server.py - Add a regression test for alias retention | 26 天前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
Supporting features for IcePop and KPop (#1405) * docs(cli): sync cli reference for icepop and kpop params * feat(examples): add icepop and kpop configs for gsm8k * style: fix end-of-file newline in icepop/kpop configs * fix(icepop/kpop): detach imp_ratio and KL inputs to prevent gradient leak * fix(kpop): use <= for KL threshold to include boundary * refactor: unify icepop/kpop into rejection_sampling with binary_kl metric - Add binary_kl metric to RejectionSamplingConfig for KPop - Remove enable_icepop/icepop_alpha/icepop_beta/enable_kpop/kpop_phi - Update gsm8k_icepop.yaml to use rejection_sampling with ratio+lower+upper - Update gsm8k_kpop.yaml to use rejection_sampling with binary_kl+upper - Sync CLI docs | 3 个月前 | |
Supporting features for IcePop and KPop (#1405) * docs(cli): sync cli reference for icepop and kpop params * feat(examples): add icepop and kpop configs for gsm8k * style: fix end-of-file newline in icepop/kpop configs * fix(icepop/kpop): detach imp_ratio and KL inputs to prevent gradient leak * fix(kpop): use <= for KL threshold to include boundary * refactor: unify icepop/kpop into rejection_sampling with binary_kl metric - Add binary_kl metric to RejectionSamplingConfig for KPop - Remove enable_icepop/icepop_alpha/icepop_beta/enable_kpop/kpop_phi - Update gsm8k_icepop.yaml to use rejection_sampling with ratio+lower+upper - Update gsm8k_kpop.yaml to use rejection_sampling with binary_kl+upper - Sync CLI docs | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
feat:enable v2 training pipeline with controller parity (#1363) * feat: enable v2 training pipeline with controller parity Bring GatewayTrainController and RolloutControllerV2 to full parity with v1 controllers for RL training paths. Key changes: - Route to RolloutControllerV2 when config._version=="v2" - Add version management, connect_engine, clear_batches to GatewayTrainController - Unify HTTP client session in GatewayTrainController (follows PR #1354) - Switch default workflow to MathAgent in example configs - Add agent config section to all example YAML files - Remove obsolete get_custom_reward_fn from reward module * fix: update wu controller connect method * chore: unblock CI for grpo and grpo_lora with admin key + lora name * chore: unblock CI for v2 parity | 3 个月前 | |
refactor(trainer): move trainer modules from experimental to areal/trainer (#896) * refactor(trainer): move trainer modules from experimental to areal/trainer Move trainer-related modules to establish a cleaner architecture: - Move PPOTrainer and SFTTrainer from areal/experimental/trainer/ to areal/trainer/ - Move PPO actor/critic from areal/engine/ppo/ to areal/trainer/ppo/ - Move SFT lm_engine from areal/engine/sft/ to areal/trainer/sft/ - Move RW engine from areal/engine/rw/ to areal/trainer/rw/ - Export PPOTrainer and SFTTrainer from top-level areal package This refactoring separates training algorithm concerns (trainer/) from backend infrastructure (engine/), making the codebase more modular. The trainers can now be imported directly via `from areal import PPOTrainer`. Updates all imports across examples, tests, docs, and internal modules. * minor fix test * fix * fix missing links | 7 个月前 | |
refactor(api): migrate allocation_mode to per-engine backend fields (#1044) * refactor(api): migrate allocation_mode to per-engine backend fields Replace the centralized `allocation_mode` string with explicit `backend` fields on `TrainEngineConfig` and `InferenceEngineConfig`. Each engine now owns its own backend+parallelism spec (e.g. `fsdp:d4`, `sglang:d4t2`), eliminating implicit auto-backend selection and the shared `AllocationMode` object. Key changes: - Add `backend` field to TrainEngineConfig and InferenceEngineConfig - Add `ModelAllocation.from_str()` for single-component parsing - Remove `AllocationMode` public export (replaced by `ModelAllocation`) - Rename internal `AllocationMode` to `_AllocationMode` for SPMD launcher backward compatibility with FutureWarning - Remove auto-backend selection — explicit backend prefix is now required - Controllers (`TrainController`, `RolloutController`) parse `backend` directly instead of receiving `alloc_mode` from trainers - `WeightUpdateMeta.alloc_mode` replaced by `gen_allocation` (single `ModelAllocation`) - Add `RWTrainer` and `ArchonRWEngine` for reward model training - Remove `get_model_update_meta()` helper (logic moved to trainers) - Update all YAML configs, examples, docs (EN+ZH), and tests BREAKING CHANGE: `AllocationMode` is removed from public API. Users must migrate to per-engine `backend` fields. SPMD launchers emit deprecation warnings. * chore(ci): fix backend specifier for vlm sft test * fix: fix bare dims for actor backends * chore(docs): fix reminder for bare allocation dims | 5 个月前 | |
fix(archon): harden FP8 blockwise training for TP and MoE scenarios (#1118) Improve FP8 robustness: extend shard alignment validation to GroupedExperts, fix DTensor handling in dense FP8 linear forward, add early checkpoint compatibility checks, and clean up config API. Key changes: - Validate GroupedExperts w1/w2/w3 shapes in post-parallelism check - Convert DTensor to local tensor in FP8 linear forward for TP>1 - Restrict FP8 dequant to float8_e4m3fn matching prepare path - Fail fast on Shard(1) FP8 checkpoints before DCP I/O - Add ArchonFP8Config.enabled property to centralize mode checks - Document exclude_modules default list in YAML example | 5 个月前 | |
feat: support Qwen3.5 hybrid AWEX training and SWE data (#1703) * feat: support Qwen3.5 hybrid AWEX training and SWE data * fix(examples): configure the data service guard for Slurm * fix: preserve older GDN runtimes and validate two-node RL recipe * fix(experimental): sync reasoning streaming and drop duplicate packed gate * fix(experimental): preserve interaction metadata and reasoning Keep dump metadata across proxy exports and preserve reasoning in both responses and cached assistant messages. Parse raw tool calls before splitting reasoning so unclosed thinking blocks retain tool calls. Key changes: - Propagate session and interaction metadata through serialization - Align streaming, non-streaming, and cached reasoning - Cover metadata dumps and Qwen tool calls with regression tests Refs: #1703 * test: align CI fixtures with reasoning and loss normalization Disable thinking in short-budget answer-delivery tests and preserve the full assistant message when continuing tool conversations. Cover reasoning parent linkage and both Megatron loss normalization modes. Refs: #1703 * refactor: remove per-token loss changes from Qwen AWEX port * ci: pin uv for package installation checks uv 0.12.14 rejects the blosc wheel because it targets the Linux virtual environment's lib64 symlink. Pin the last known-good uv release so package installation remains reproducible. --------- Co-authored-by: dingzhiqiang <dingzhiqiang@users.noreply.github.com> Co-authored-by: chucai.dzq <chucai.dzq@alibaba-inc.com> Co-authored-by: sitabulaixizawaluduo <ljl2020110773@gmail.com> | 1 天前 |
Hyper-parameters for GSM8K Finetuning on Qwen2.5-1.5b-Instruct
gsm8k_grpo.yaml uses 6e-6, retuned for the current FSDP FP32-master optimizer path.
Current FP32-Master Recipe
A seed-1 run on 8 NVIDIA A800 GPUs completed the official 10-epoch schedule with the following held-out evaluation results:
| lr | weight decay | group size | best eval reward | final eval reward |
|---|---|---|---|---|
| 6.0E-6 | 0.017 | 4 | 0.78412 | 0.77767 |
Historical Pre-FP32-Master Sweep
The results below were collected with BF16 parameter and optimizer-state storage. They are retained for reference and are not directly comparable with the current recipe.
| lr | weight decay | group size | max task_reward |
|---|---|---|---|
| 1.70E-05 | 0.017 | 4 | 0.79570 |
| 1.30E-05 | 0.015 | 8 | 0.79355 |
| 1.50E-05 | 0.01 | 4 | 0.79043 |
| 1.50E-05 | 0.02 | 4 | 0.78984 |
| 1.00E-05 | 0.02 | 4 | 0.78311 |
| 1.00E-05 | 0.01 | 8 | 0.78066 |
Training Details
- Devices: 8 Nvidia H800 GPUs
- Optimizer: Adam
- LR Scheduler: Constant
- Gradient Clipping: 1.0
- Max_new_tokens: 1024
- Max_head_offpolicyness: 2
- Training Time: ~35 minutes (batchsize 4), ~65 minutes (batchsize 8)