| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat:Add chinese doc (#969) * fix(docs): add language toggle button with proper styling and icon - Add lang-toggle.js and lang-toggle.css to en/zh _static directories - Update _config.yml to reference correct paths - Modify build_all.sh to copy static files and figures - Fix URL redirect logic (remove trailing slash) - Style button with translate icon, gray theme matching nav icons - Dynamically adjust icon size based on existing header icons * fix(docs): add language toggle button with proper styling and icon - Add lang-toggle.js and lang-toggle.css to en/zh _static directories - Update _config.yml to reference correct paths - Modify build_all.sh to copy static files and figures - Fix URL redirect logic (remove trailing slash) - Style button with translate icon, gray theme matching nav icons - Dynamically adjust icon size based on existing header icons * Update docs/build_all.sh Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update docs/_static/js/lang-toggle.js Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 5 个月前 | |
feat(distillation): add on-policy distillation using RolloutEngine (#1376) * feat(distillation): add on-policy distillation using RolloutEngine * fix: add build_score_request function to vllm_remote.py * chore(pre-commit): apply formatting suggestions * fix: pass pp_size to SGLangConfig and add defensive checks for rollout responses * chore(config): warn when multiple teacher engine types are configured | 2 个月前 | |
feat(trainer): add dpo (#1190) * feat(trainer): add DPO trainer with FSDP backend Add Direct Preference Optimization (Rafailov et al. 2023) as a new trainer. The policy is directly optimized to prefer chosen over rejected responses via a contrastive loss on log-probability ratios against a frozen reference model, removing the need for a separately trained reward model. Reference logprobs are computed online each step by a colocated ref engine, following the PPO/GRPO pattern. FSDP is the supported backend; Megatron and Archon variants raise NotImplementedError as placeholders. Verified on Qwen2.5-7B-Base + Anthropic/hh-rlhf (1 epoch, no SFT): reward_accuracy rises from 0.50 to ~0.70 and reward_margin grows monotonically, matching the original DPO paper's HH-RLHF results. * fix(trainer): fix DPO config forwarding, require ref model, and correct IPO normalization Fixes several issues found during PR review of the DPO trainer: Key changes: - Create DPOEngineConfig(TrainEngineConfig) embedding beta and loss_type, fixing silent parameter drop in single-controller mode (as_controller never forwarded beta/loss_type to workers) - Make ref a required field in DPOConfig (ref_logprobs are required at runtime, so config should enforce this upfront) - Remove zero-ref fallback in compute_dpo_loss; use input_["ref_logprobs"] directly - Add IPO loss with per-token length normalization matching TRL author- confirmed convention (normalize per-sequence logratios by completion length before the squared loss) - Remove all ref-is-None guard branches from DPOTrainer - Update docs, YAML config, and tests for all changes Refs: #1190 --------- Co-authored-by: 博惟 <bowei.fw@antgroup.com> | 4 个月前 | |
feat(trainer): add flexible GAE lambda strategies (#1600) * feat: add turn-level GAE support Treat each generated turn as a GAE timestep while preserving the token-level default and token-local KL regularization. Key changes: - propagate and validate token-aligned turn IDs - compute turn-level advantages without full CPU sequence scans - filter structural metadata at FSDP and Archon model boundaries - document the new selector and add focused regression tests * feat(trainer): add dynamic per-sample GAE lambda Allow GAE lambda to vary by trajectory using effective token or turn lengths while preserving static float behavior. Key changes: - Resolve custom lambda functions and keyword arguments from config - Add VAPO length-adaptive GAE with empty-trajectory handling - Validate per-sample lambda tensors and cover token and turn modes * perf(trainer): reduce GAE preprocessing overhead Hoist loop-invariant tensor work and bypass dynamic trajectory length construction when GAE lambda is static. * feat(trainer): add relative-position GAE lambda * fix(trainer): allow token lambda without turn metadata Keep custom token-level GAE lambda functions compatible with rollout workflows that do not emit turn IDs, while preserving the metadata requirement for turn-level GAE. * docs: regenerate CLI reference for GAE options Keep the generated configuration reference aligned with the current main branch after porting the AntCode GAE changes. * docs: document flexible GAE configuration Explain token- and turn-level recurrences, KL and critic semantics, dynamic lambda strategies, and custom workflow turn IDs in English and Chinese.\n\nFix CLI default rendering for dataclass factories and cover it with unit tests. * refactor(trainer): extract GAE helpers Keep PPOActor focused on training orchestration by moving GAE kernels, turn metadata validation, and lambda context construction into a dedicated module. --------- Co-authored-by: Wenhao Zhou <miumiu.zwh@antgroup.com> | 16 天前 | |
docs: clarify terminology in the M2PO guides (#1635) Signed-off-by: shicyu <77009832+Wrestle7@users.noreply.github.com> | 12 天前 | |
docs: fix broken LaTeX rendering in bilingual algorithm docs (#1004) Fix incorrect LaTeX escaping and math block formatting that caused rendering issues in docs/en and docs/zh algorithm pages. Key changes: - normalize LaTeX/math syntax in GRPO, M2PO, prox_approx, and distillation docs - add missing MyST anchors/refs and align related tutorial links - include zh distillation chapter and TOC entries to keep en/zh docs in sync - update pre-commit mdformat exclusions and agent docs build guidance | 5 个月前 | |
feat(distillation): add on-policy distillation using RolloutEngine (#1376) * feat(distillation): add on-policy distillation using RolloutEngine * fix: add build_score_request function to vllm_remote.py * chore(pre-commit): apply formatting suggestions * fix: pass pp_size to SGLangConfig and add defensive checks for rollout responses * chore(config): warn when multiple teacher engine types are configured | 2 个月前 | |
feat(utils): add Karmarkar-Karp partitioning algorithm for sequence packing (#1151) Add KK (Largest Differencing Method) as an alternative to FFD for micro-batch allocation. KK produces more balanced partitions with lower max-min spread, beneficial for RL workloads with variable sequence lengths. Key changes: - Add _KKSet, _KKState, _kk_partition, kk_allocate in seqpack.py - Add packing_algorithm field to MicroBatchSpec (ffd/kk) - Wire KK allocation through dist_rollout and data utils - Add sequence_packing docs (en/zh) and CLI reference updates - Add comprehensive unit tests and torchrun benchmark Refs: #1151 | 4 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 5 个月前 | ||
| 2 个月前 | ||
| 4 个月前 | ||
| 16 天前 | ||
| 12 天前 | ||
| 5 个月前 | ||
| 2 个月前 | ||
| 4 个月前 |
