Closed
[Bug-Report|缺陷反馈]: 910B2 grouped_matmul_finalize_routing算子开确定性之后卡死 #4334
yangchao888Add on  Aug 14Closed at  Aug 25
yangchao888
yangchao888
Aug 14 Create

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

910B2 grouped_matmul_finalize_routing算子开确定性之后卡死

Environment / 环境信息 (Mandatory / 必填)

芯片型号:Ascend910B2
CANN 版本:9.0.0

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

输入信息:
x_int8.shape: torch.Size([16548, 1536]), x_int8.dtype: torch.int8
weight_int8.shape: torch.Size([180, 1536, 1536]), weight_int8.dtype: torch.int8
group_list.shape: torch.Size([180]), group_list.dtype: torch.int64
scale.shape: torch.Size([180, 1536]), scale.dtype: torch.float32
pertoken_scale.shape: torch.Size([16548]), pertoken_scale.dtype: torch.float32
shared_input.shape: torch.Size([394, 1536]), shared_input.dtype: torch.bfloat16
logit.shape: torch.Size([16548]), logit.dtype: torch.float32
row_index.shape: torch.Size([16548]), row_index.dtype: torch.int32
output_bs: 394
group_list_type: 1
开确定性:
torch.use_deterministic_algorithms(True)
output = torch_npu.npu_grouped_matmul_finalize_routing(
x_int8,
weight_int8,
group_list,
scale=scale,
pertoken_scale=pertoken_scale,
shared_input=shared_input,
logit=logit,
row_index=row_index,
output_bs=output_bs,
group_list_type=group_list_type,
)
调用output = torch_npu.npu_grouped_matmul_finalize_routing()后,一直卡住无返回。
torch.use_deterministic_algorithms(True) 改为 torch.use_deterministic_algorithms(False) 后正常返回。

Describe the expected behavior / 预期结果 (Mandatory / 必填)

调用output = torch_npu.npu_grouped_matmul_finalize_routing()后,正常输出结果

调用output = torch_npu.npu_grouped_matmul_finalize_routing()后,一直卡住无返回

Special notes for this issue/备注 (Optional / 选填)

likedislike
weihao18
weihao18Collaborators
Aug 14 Comments:
likedislike
CANN-robotCANN-robotCollaborators
Aug 14 Assign hunter_li as responsible person
CANN-robotCANN-robotCollaborators
Aug 14 Assign zhangzhuoran3 as responsible person, remove original responsible person: hunter_li
zhangzhuoran3zhangzhuoran3Collaborators
Aug 19 Associated pull request
CANN-robotCANN-robotCollaborators
Aug 25 Close issue
CANN-robotCANN-robotCollaborators
Aug 25 Added label: resolved