已开启
[Bug]: attn input layout为TND时,pynative模式下由于缺少actual_seq_len,导致推理失败 #2602
Chao创建于  5 天前
Chao
Chao
5 天前 创建

Checklist

🐞 问题详细描述

Pynative模式下

模型配置:

image.png

use_ring_attention默认为false,此时input_layout会被赋值为TND。

image.png

模型前向由于缺少actual_seq_len,导致如下报错:

Traceback (most recent call last):
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/run_mindformer.py", line 263, in <module>
    main(args_)
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/run_mindformer.py", line 72, in main
    trainer.inference()
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/trainer/trainer.py", line 1172, in inference
    sample_results = generate_from_records(
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/trainer/inference.py", line 562, in generate_from_records
    generated = generate_batch(
                ^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/trainer/inference.py", line 324, in generate_batch
    logits = model[0](input_ids=input_ids, position_ids=position_ids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/site-packages/mindspore/nn/cell.py", line 1420, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/models/deepseek3/modeling_deepseek_v3_pynative.py", line 67, in construct
    return self.model(
           ^^^^^^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/site-packages/mindspore/nn/cell.py", line 1420, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/base_models/gpt/gpt_model.py", line 355, in construct
    hidden_states, rotary_pos_emb, mscale = self.language_model(
                                            ^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/base_models/gpt/gpt_model.py", line 493, in language_model
    hidden_states = self.decoder(
                    ^^^^^^^^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/site-packages/mindspore/nn/cell.py", line 1420, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/transformers/transformer_block.py", line 377, in construct
    hidden_states, _ = layer(
                       ^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/site-packages/mindspore/nn/cell.py", line 1420, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/transformers/transformer_layer.py", line 234, in construct
    attn_result = self.self_attention(
                  ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.13/lib/python3.11/site-packages/mindspore/nn/cell.py", line 1420, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/hpfs/huawei/lvchao/code/TeleChat4-285B-logits-train/mindformers/mindformers/pynative/transformers/multi_latent_attention.py", line 218, in construct
    raise ValueError("TND attention requires actual_seq_len.")
ValueError: TND attention requires actual_seq_len.

详细的环境信息描述

硬件环境:910C

软件版本:
CANN

软件名 版本
CANN 9.1.0
Python 3.11.13
mindformers v2.1.0

其他辅助信息

版本信息

不涉及

Thanks for contributing 🎉!

likedislike
ChaoChao
5 天前 添加了label:bug
ChaoChao
5 天前 关联组件 由 [] 改变为 [B-SIG-MindFormers-Infer]
ChaoChao
5 天前 问题重现类型 由 [] 改变为 [有条件必然重现]
ChaoChao
5 天前 关联问题类型 由 [] 改变为 [功能]
ChaoChao
5 天前 关联分支 由 [] 改变为 [r2.10]
ChaoChao
5 天前 问题后端类型 由 [] 改变为 [Ascend]
ChaoChao
5 天前 关联了pull request:fix(pynative): add actual_seq_len when input layout is TND in predict run mode