| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
docs: streamline CLI usage and reorganize tutorials (#878) * docs: streamline CLI usage and reorganize tutorials Reorganize documentation structure and remove deprecated eval allocation mode: Documentation: - Move gsm8k_grpo tutorial from docs/lite/ to docs/tutorial/ - Remove redundant docs/lite/ directory (boba_grpo_vllm_npu.md, gsm8k_grpo.md) - Streamline quickstart.md with simplified CLI examples - Rewrite eval.md with clearer structure and examples - Clean up algorithm docs (remove empty ppo.md, fix cross-references) API cleanup: - Remove DECOUPLED_EVAL allocation type and eval expression parsing - Remove EvalType and EvalAllocationExpression classes - Simplify allocation grammar by removing eval-related rules Examples: - Update gsm8k_eval.py with improved evaluation workflow - Fix CLI examples across READMEs and example scripts - Update geometry3k_grpo.sh for NPU workflow * fix(docs): update broken links to gsm8k_grpo tutorial Update relative links to gsm8k_grpo.md after moving from docs/lite/ to docs/tutorial/ directory. * fix(api): remove unused 'cpu' backend from ModelAllocation The 'cpu' backend was listed in the Literal type but never referenced in the grammar, parser, or allocation logic. * fix legacy links | 6 个月前 | |
refactor(api): migrate allocation_mode to per-engine backend fields (#1044) * refactor(api): migrate allocation_mode to per-engine backend fields Replace the centralized `allocation_mode` string with explicit `backend` fields on `TrainEngineConfig` and `InferenceEngineConfig`. Each engine now owns its own backend+parallelism spec (e.g. `fsdp:d4`, `sglang:d4t2`), eliminating implicit auto-backend selection and the shared `AllocationMode` object. Key changes: - Add `backend` field to TrainEngineConfig and InferenceEngineConfig - Add `ModelAllocation.from_str()` for single-component parsing - Remove `AllocationMode` public export (replaced by `ModelAllocation`) - Rename internal `AllocationMode` to `_AllocationMode` for SPMD launcher backward compatibility with FutureWarning - Remove auto-backend selection — explicit backend prefix is now required - Controllers (`TrainController`, `RolloutController`) parse `backend` directly instead of receiving `alloc_mode` from trainers - `WeightUpdateMeta.alloc_mode` replaced by `gen_allocation` (single `ModelAllocation`) - Add `RWTrainer` and `ArchonRWEngine` for reward model training - Remove `get_model_update_meta()` helper (logic moved to trainers) - Update all YAML configs, examples, docs (EN+ZH), and tests BREAKING CHANGE: `AllocationMode` is removed from public API. Users must migrate to per-engine `backend` fields. SPMD launchers emit deprecation warnings. * chore(ci): fix backend specifier for vlm sft test * fix: fix bare dims for actor backends * chore(docs): fix reminder for bare allocation dims | 5 个月前 | |
feat(api): add unified RejectionSamplingConfig for async training (#1088) Replace behave_imp_weight_cap/behave_imp_weight_mode with unified RejectionSamplingConfig supporting multiple metrics (ratio, kl_k1, kl_k2, kl_k3), levels (token/sequence), and actions (mask/clamp). Key changes: - Add RejectionSamplingConfig dataclass with comprehensive validation - Implement apply_rejection_sampling for 1D packed and 2D padded formats - Fix loss denominator scaling bug in mask mode (save count before filtering) - Use geometric mean for sequence-level ratio aggregation (matching GSPO) - Broadcast sequence-level geometric mean as uniform behave_imp_weight - Warn when use_decoupled_loss=True but rejection_sampling is None - Update ppo_actor_loss_fn and grpo_loss_fn to use new config - Migrate 40 example configs to new rejection_sampling field - Add 43 unit tests covering all modes, metrics, and edge cases Refs: #1052 | 4 个月前 | |
refactor(api): migrate allocation_mode to per-engine backend fields (#1044) * refactor(api): migrate allocation_mode to per-engine backend fields Replace the centralized `allocation_mode` string with explicit `backend` fields on `TrainEngineConfig` and `InferenceEngineConfig`. Each engine now owns its own backend+parallelism spec (e.g. `fsdp:d4`, `sglang:d4t2`), eliminating implicit auto-backend selection and the shared `AllocationMode` object. Key changes: - Add `backend` field to TrainEngineConfig and InferenceEngineConfig - Add `ModelAllocation.from_str()` for single-component parsing - Remove `AllocationMode` public export (replaced by `ModelAllocation`) - Rename internal `AllocationMode` to `_AllocationMode` for SPMD launcher backward compatibility with FutureWarning - Remove auto-backend selection — explicit backend prefix is now required - Controllers (`TrainController`, `RolloutController`) parse `backend` directly instead of receiving `alloc_mode` from trainers - `WeightUpdateMeta.alloc_mode` replaced by `gen_allocation` (single `ModelAllocation`) - Add `RWTrainer` and `ArchonRWEngine` for reward model training - Remove `get_model_update_meta()` helper (logic moved to trainers) - Update all YAML configs, examples, docs (EN+ZH), and tests BREAKING CHANGE: `AllocationMode` is removed from public API. Users must migrate to per-engine `backend` fields. SPMD launchers emit deprecation warnings. * chore(ci): fix backend specifier for vlm sft test * fix: fix bare dims for actor backends * chore(docs): fix reminder for bare allocation dims | 5 个月前 | |
refactor: simplifying the implementation of customized workflow with context management (#785) * support rollout dumping in workflow executor * consolidate config and high-level API changes * use workflow context to pass stats scope * fix test * fix test_examples * use keyword arguments in submit * rename `eval` to `is_eval` to avoid conflict with python eval func * Update areal/experimental/trainer/rl.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> | 7 个月前 |
Training a Multi-Turn GSM8K Math Agent in AReaL
Files in this folder presents an example that train a multi-turn GSM8K math agent from
Qwen/Qwen2.5-1.5B-Instruct, using ArealOpenAI APIs and its concat mode to organize
training data and discount reward.
To run the example
python3 examples/multi_turn_math/gsm8k_rl_mt.py \
--config examples/multi_turn_math/gsm8k_grpo_mt.yaml \
scheduler.type=ray \
experiment_name=gsm8k-grpo-multiturn trial_name=trial0
only the following config are added compared to the original gsm8k_grpo.yaml config:
export_style: concat
agent_run_args:
max_turns: 2
Reward Curve
