已合并
[BUGFIX!] 增加legacy分支下选择性重计算相关说明及warning提示 #2141
EX_mitsuX创建于 2025年4月1日
[BUGFIX!] 增加legacy分支下选择性重计算相关说明及warning提示 #2141
已合并
从refs/pull/2141/head合入到core_r0.8.0
共 2 个文件变更+5-1
| @@ -27,6 +27,7 @@ | |||
| 27 | 同时配置`--recompute-activations` 、`--recompute-granularity full`生效选择性重计算。 | 27 | 同时配置`--recompute-activations` 、`--recompute-granularity full`生效选择性重计算。 |
| 28 | 28 | ||
| 29 | 当脚本配置了`--recompute-method block`、`--recompute-granularity full`、`--num-layers-per-virtual-pipeline-stage N`参数时，用户可以通过`--recompute-num-layers N`参数来配置每个vpp stage做多少层重计算，参数`--enable-recompute-layers-per-pp-rank`可用于修改此情况下`--recompute-num-layers N`参数的语义，新的语义表示无视vpp，按每个pp stage来配置重计算层数。 | 29 | 当脚本配置了`--recompute-method block`、`--recompute-granularity full`、`--num-layers-per-virtual-pipeline-stage N`参数时，用户可以通过`--recompute-num-layers N`参数来配置每个vpp stage做多少层重计算，参数`--enable-recompute-layers-per-pp-rank`可用于修改此情况下`--recompute-num-layers N`参数的语义，新的语义表示无视vpp，按每个pp stage来配置重计算层数。 |
| 30 | +注意：在legacy分支下，开启--use-flash-attn将无法使用选择性重计算。 | ||
| 30 | 31 | ||
| 31 | ## 使用影响 | 32 | ## 使用影响 |
| 32 | 显存开销降低、性能降低。 | 33 | 显存开销降低、性能降低。 |
| @@ -552,7 +552,10 @@ def validate_args_wrapper(validate_args): | |||
| 552 | args.use_fusion_attn_v2 = True | 552 | args.use_fusion_attn_v2 = True |
| 553 | if args.use_fusion_attn_v2: | 553 | if args.use_fusion_attn_v2: |
| 554 | args.use_flash_attn = True | 554 | args.use_flash_attn = True |
| 555 | - print("[WARNING] \"use_fusion_attn_v2\" is not recommended. This feature is not officially released.") | 555 | + print("[WARNING] \"use_fusion_attn_v2\" is not recommended. This feature is not officially released.") |
| 556 | + if args.use_flash_attn and args.use_legacy_models: | ||
| 557 | + if args.recompute_granularity == "selective": | ||
| 558 | + print("[WARNING] In legacy models, \"--recompute-activations\" is not recommended. This feature is not currently supported.") | ||
| 556 | 559 | ||
| 557 | # for vpp assert pp should > 2 | 560 | # for vpp assert pp should > 2 |
| 558 | flag_num_layers_per_virtual_pipeline_stage = None | 561 | flag_num_layers_per_virtual_pipeline_stage = None |