| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
perf(engine): stream Megatron microbatches from CPU (#1622) * perf(engine): stream Megatron microbatches from CPU Keep full training batches off the accelerator and transfer only the current microbatch before forward. This prevents packed and padded batch representations from scaling GPU residency with the global batch size. Key changes: - Broadcast Megatron batch RPC payloads through CPU process groups - Stage trainer and engine microbatch construction on CPU - Preserve tree, VLM, and loss-weight behavior during lazy transfer - Add CPU streaming and virtual-pipeline iterator regression tests * fix: keep streamed training payloads off accelerator Avoid staging complete v2 training payloads on the accelerator before Megatron microbatch streaming. Replace tree-count object collectives with a backend-compatible scalar MAX reduction and cover both placement paths with unit tests. * fix: reduce CPU-staged stats on compatible devices CPU-staged Megatron payloads can leave rollout statistics on CPU while the data-parallel process group uses NCCL or HCCL. Move only locally aggregated scalars to the backend-compatible device before collectives and keep empty reward-model placeholders on the engine device. Key changes: - add backend-aware scalar all-reduce handling - align empty reward-model stats with the engine device - cover CPU stats and empty train/eval batches Refs: #1622 * fix: harden streamed microbatch distributed coverage Fail fast when CPU-staged RPC methods lack their Gloo mirror and preserve tuple containers during recursive tensor broadcast. Extend distributed training coverage across TP1/PP1 and TP2/PP2 with multi-microbatch gradient-norm parity checks. --------- Co-authored-by: daihao <dh183333@antgroup.com> | 18 天前 | |
perf: reduce duplicated VLM rollout processing and transfers (#1671) * perf(infra): add group-scoped processor cache * perf(workflow): reuse processor results across VLM rollout groups * perf(experimental): cache agent multimodal prompt processing * perf(experimental): share processor cache across proxy sessions * perf(experimental): add grouped multimodal tensor references * perf(experimental): share tensor references across proxy sessions * perf(infra): preserve shared RTensor aliases * perf(infra): preserve aliases across staged engine RPCs * perf(trainer): avoid multimodal advantage RPC transfers * test(experimental): complete multimodal export fixtures * fix(experimental): prevent shared tensor refs from surviving store reuse * fix(infra): cancel rollout siblings before group finalization * fix(infra): retain source tensors during alias-preserving remotize | 9 天前 | |
perf(engine): stream Megatron microbatches from CPU (#1622) * perf(engine): stream Megatron microbatches from CPU Keep full training batches off the accelerator and transfer only the current microbatch before forward. This prevents packed and padded batch representations from scaling GPU residency with the global batch size. Key changes: - Broadcast Megatron batch RPC payloads through CPU process groups - Stage trainer and engine microbatch construction on CPU - Preserve tree, VLM, and loss-weight behavior during lazy transfer - Add CPU streaming and virtual-pipeline iterator regression tests * fix: keep streamed training payloads off accelerator Avoid staging complete v2 training payloads on the accelerator before Megatron microbatch streaming. Replace tree-count object collectives with a backend-compatible scalar MAX reduction and cover both placement paths with unit tests. * fix: reduce CPU-staged stats on compatible devices CPU-staged Megatron payloads can leave rollout statistics on CPU while the data-parallel process group uses NCCL or HCCL. Move only locally aggregated scalars to the backend-compatible device before collectives and keep empty reward-model placeholders on the engine device. Key changes: - add backend-aware scalar all-reduce handling - align empty reward-model stats with the engine device - cover CPU stats and empty train/eval batches Refs: #1622 * fix: harden streamed microbatch distributed coverage Fail fast when CPU-staged RPC methods lack their Gloo mirror and preserve tuple containers during recursive tensor broadcast. Extend distributed training coverage across TP1/PP1 and TP2/PP2 with multi-microbatch gradient-norm parity checks. --------- Co-authored-by: daihao <dh183333@antgroup.com> | 18 天前 | |
perf(engine): stream Megatron microbatches from CPU (#1622) * perf(engine): stream Megatron microbatches from CPU Keep full training batches off the accelerator and transfer only the current microbatch before forward. This prevents packed and padded batch representations from scaling GPU residency with the global batch size. Key changes: - Broadcast Megatron batch RPC payloads through CPU process groups - Stage trainer and engine microbatch construction on CPU - Preserve tree, VLM, and loss-weight behavior during lazy transfer - Add CPU streaming and virtual-pipeline iterator regression tests * fix: keep streamed training payloads off accelerator Avoid staging complete v2 training payloads on the accelerator before Megatron microbatch streaming. Replace tree-count object collectives with a backend-compatible scalar MAX reduction and cover both placement paths with unit tests. * fix: reduce CPU-staged stats on compatible devices CPU-staged Megatron payloads can leave rollout statistics on CPU while the data-parallel process group uses NCCL or HCCL. Move only locally aggregated scalars to the backend-compatible device before collectives and keep empty reward-model placeholders on the engine device. Key changes: - add backend-aware scalar all-reduce handling - align empty reward-model stats with the engine device - cover CPU stats and empty train/eval batches Refs: #1622 * fix: harden streamed microbatch distributed coverage Fail fast when CPU-staged RPC methods lack their Gloo mirror and preserve tuple containers during recursive tensor broadcast. Extend distributed training coverage across TP1/PP1 and TP2/PP2 with multi-microbatch gradient-norm parity checks. --------- Co-authored-by: daihao <dh183333@antgroup.com> | 18 天前 | |
feat(trainer): add dpo (#1190) * feat(trainer): add DPO trainer with FSDP backend Add Direct Preference Optimization (Rafailov et al. 2023) as a new trainer. The policy is directly optimized to prefer chosen over rejected responses via a contrastive loss on log-probability ratios against a frozen reference model, removing the need for a separately trained reward model. Reference logprobs are computed online each step by a colocated ref engine, following the PPO/GRPO pattern. FSDP is the supported backend; Megatron and Archon variants raise NotImplementedError as placeholders. Verified on Qwen2.5-7B-Base + Anthropic/hh-rlhf (1 epoch, no SFT): reward_accuracy rises from 0.50 to ~0.70 and reward_margin grows monotonically, matching the original DPO paper's HH-RLHF results. * fix(trainer): fix DPO config forwarding, require ref model, and correct IPO normalization Fixes several issues found during PR review of the DPO trainer: Key changes: - Create DPOEngineConfig(TrainEngineConfig) embedding beta and loss_type, fixing silent parameter drop in single-controller mode (as_controller never forwarded beta/loss_type to workers) - Make ref a required field in DPOConfig (ref_logprobs are required at runtime, so config should enforce this upfront) - Remove zero-ref fallback in compute_dpo_loss; use input_["ref_logprobs"] directly - Add IPO loss with per-token length normalization matching TRL author- confirmed convention (normalize per-sequence logratios by completion length before the squared loss) - Remove all ref-is-None guard branches from DPOTrainer - Update docs, YAML config, and tests for all changes Refs: #1190 --------- Co-authored-by: 博惟 <bowei.fw@antgroup.com> | 4 个月前 | |
fix(infra): prevent leaks after RTensor delete failures (#1633) * fix(infra): prevent leaks after RTensor delete failures Preserve failed shard IDs across one clear_batches call so transient storage-node outages can recover. Stop training after a second failure instead of silently accumulating CPU tensors. Key changes: - send DELETE payloads through the existing bounded retry helper - surface per-node failures and storage cleanup statistics - retry failed shards across one step, then fail after worker cleanup - preserve the whole pending batch atomically on cancellation Refs: #1581 * fix(infra): make batch cleanup failure-safe Keep exhausted RTensor shards pending until worker buffers drain, so a worker RPC error cannot erase storage leak state. Run every trainer role cleanup before propagating the first failure. Refs: #1581 | 18 天前 | |
fix(infra): prevent leaks after RTensor delete failures (#1633) * fix(infra): prevent leaks after RTensor delete failures Preserve failed shard IDs across one clear_batches call so transient storage-node outages can recover. Stop training after a second failure instead of silently accumulating CPU tensors. Key changes: - send DELETE payloads through the existing bounded retry helper - surface per-node failures and storage cleanup statistics - retry failed shards across one step, then fail after worker cleanup - preserve the whole pending batch atomically on cancellation Refs: #1581 * fix(infra): make batch cleanup failure-safe Keep exhausted RTensor shards pending until worker buffers drain, so a worker RPC error cannot erase storage leak state. Run every trainer role cleanup before propagating the first failure. Refs: #1581 | 18 天前 | |
fix(infra): prevent leaks after RTensor delete failures (#1633) * fix(infra): prevent leaks after RTensor delete failures Preserve failed shard IDs across one clear_batches call so transient storage-node outages can recover. Stop training after a second failure instead of silently accumulating CPU tensors. Key changes: - send DELETE payloads through the existing bounded retry helper - surface per-node failures and storage cleanup statistics - retry failed shards across one step, then fail after worker cleanup - preserve the whole pending batch atomically on cancellation Refs: #1581 * fix(infra): make batch cleanup failure-safe Keep exhausted RTensor shards pending until worker buffers drain, so a worker RPC error cannot erase storage leak state. Run every trainer role cleanup before propagating the first failure. Refs: #1581 | 18 天前 | |
fix(infra): prevent leaks after RTensor delete failures (#1633) * fix(infra): prevent leaks after RTensor delete failures Preserve failed shard IDs across one clear_batches call so transient storage-node outages can recover. Stop training after a second failure instead of silently accumulating CPU tensors. Key changes: - send DELETE payloads through the existing bounded retry helper - surface per-node failures and storage cleanup statistics - retry failed shards across one step, then fail after worker cleanup - preserve the whole pending batch atomically on cancellation Refs: #1581 * fix(infra): make batch cleanup failure-safe Keep exhausted RTensor shards pending until worker buffers drain, so a worker RPC error cannot erase storage leak state. Run every trainer role cleanup before propagating the first failure. Refs: #1581 | 18 天前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 18 天前 | ||
| 9 天前 | ||
| 18 天前 | ||
| 18 天前 | ||
| 4 个月前 | ||
| 18 天前 | ||
| 18 天前 | ||
| 18 天前 | ||
| 18 天前 |