已合并
Readme修改 #1505
AtomGit-Bot创建于 2024年11月27日
Readme修改 #1505
已合并
从refs/pull/1505/head合入到master
共 4 个文件变更+10-1
| @@ -46,4 +46,7 @@ top1 多种并行方式 提速效果： | |||
| 46 | 同时开启此优化减少显存占用3%： | 46 | 同时开启此优化减少显存占用3%： |
| 47 | <p align="center"> <img src="../../../sources/images/moe_dynamic_padding_e.png" height="500px" width="800px"></p> | 47 | <p align="center"> <img src="../../../sources/images/moe_dynamic_padding_e.png" height="500px" width="800px"></p> |
| 48 | 48 | ||
| 49 | +## 注意事项： | ||
| 50 | + | ||
| 51 | +暂不兼容 流水线并行特性，即需满足--pipeline_model_parallel_size <= 1。 | ||
| 49 | 52 | ||
| @@ -18,6 +18,8 @@ | |||
| 18 | 18 | ||
| 19 | 同时需要确保`--num-experts`能被`tp * ep`整除。 | 19 | 同时需要确保`--num-experts`能被`tp * ep`整除。 |
| 20 | 20 | ||
| 21 | +当前该特性不支持Moe Token drop and pad模式，即`--moe-expert-capacity-factor`需要为None。 | ||
| 22 | + | ||
| 21 | ## 适用场景 | 23 | ## 适用场景 |
| 22 | 24 | ||
| 23 | 细粒度小专家，类DeepSeek-V2模型，每个专家的参数量较小。 | 25 | 细粒度小专家，类DeepSeek-V2模型，每个专家的参数量较小。 |
| @@ -10,4 +10,5 @@ | |||
| 10 | | 【Prototype】Ascend Megatron MoE Alltoall Dispatcher 性能优化 | [link](megatron-moe-alltoall-dispatcher.md) | | 10 | | 【Prototype】Ascend Megatron MoE Alltoall Dispatcher 性能优化 | [link](megatron-moe-alltoall-dispatcher.md) | |
| 11 | | 【Prototype】Megatron MoE Zero Memory 性能优化 | [link](megatron-moe-zero-memory.md) | | 11 | | 【Prototype】Megatron MoE Zero Memory 性能优化 | [link](megatron-moe-zero-memory.md) | |
| 12 | | 【Prototype】Megatron MoE Allgather Overlap Dispatcher 性能优化 | [link](megatron-moe-allgather-overlap-comm.md) | | 12 | | 【Prototype】Megatron MoE Allgather Overlap Dispatcher 性能优化 | [link](megatron-moe-allgather-overlap-comm.md) | |
| 13 | -| 【Prototype】Megatron MoE Alltoall Overlap Dispatcher 性能优化 | [link](megatron-moe-alltoall-overlap-comm.md) | | 13 | +| 【Prototype】Megatron MoE Alltoall Overlap Dispatcher 性能优化 | [link](megatron-moe-alltoall-overlap-comm.md) | |
| 14 | +| 【Prototype】Megatron MoE TP拓展EP | [link](megatron-moe-tp-extend-ep.md) | | ||
| @@ -510,6 +510,9 @@ def validate_args_wrapper(validate_args): | |||
| 510 | assert num_layers_per_pipeline_stage // args.num_layers_per_virtual_pipeline_stage > 1, \ | 510 | assert num_layers_per_pipeline_stage // args.num_layers_per_virtual_pipeline_stage > 1, \ |
| 511 | 'considering args of num_layers and pipeline_model_parallel_size, vpp setting should be meaningful' | 511 | 'considering args of num_layers and pipeline_model_parallel_size, vpp setting should be meaningful' |
| 512 | 512 | ||
| 513 | + # deepspeed dropless does not support pp | ||
| 514 | + if args.moe_no_drop and args.pipeline_model_parallel_size > 1: | ||
| 515 | + raise AssertionError("--moe-no-drop is not compatible with pp") | ||
| 513 | 516 | ||
| 514 | if args.use_fused_rmsnorm: | 517 | if args.use_fused_rmsnorm: |
| 515 | if args.normalization != "RMSNorm": | 518 | if args.normalization != "RMSNorm": |