[Daily-QA|每日监测] ascend950 算子测试失败: gemm

已关闭

[Daily-QA|每日监测] ascend950 算子测试失败: gemm_ex (2026-06-29) #231

wangzitao创建于 6月29日关闭于 7月7日

wangzitao成员

6月29日创建

Describe the current behavior / 问题描述

每日质量监测发现 ascend950 全量测试失败：gemm_ex。

Environment / 环境信息

硬件: Ascend 950PR/950DT
OS: Ubuntu x86_64
CANN: 9.1.0 (build 20260625000329957)
ops-blas commit: 3a6b990841b7 (master)
运行命令: bash build.sh --soc=ascend950 --run

Steps to reproduce / 重现步骤

source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh
cd ops-blas && git fetch upstream master && git checkout master && git reset --hard upstream/master
bash build.sh --soc=ascend950 --run

Expected behavior / 预期结果

全部算子测试通过（Skipped 为 SOC 不支持，不计入失败）。

算子	状态	退出码
gemm_ex	FAIL	1

build.log 末尾 200 行

[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L0_16
[TC_L0_16] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L0_16] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L0_16] PASSED (exact match, 0/131328 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L0_16 (17 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L0_17
[TC_L0_17] Output: 1.000000 513.000000 514.000000 1025.000000 1026.000000 1027.000000 1537.000000 1538.000000 1539.000000 1540.000000 ...
[TC_L0_17] Golden: 1.000000 513.000000 514.000000 1025.000000 1026.000000 1027.000000 1537.000000 1538.000000 1539.000000 1540.000000 ...
[TC_L0_17] PASSED (exact match, 0/131328 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L0_17 (17 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_01
[TC_L1_01] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 10.000000 11.000000 ...
[TC_L1_01] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 10.000000 11.000000 ...
[TC_L1_01] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_01 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_02
[TC_L1_02] Output: 1.000000 9.000000 10.000000 17.000000 18.000000 19.000000 25.000000 26.000000 27.000000 28.000000 ...
[TC_L1_02] Golden: 1.000000 9.000000 10.000000 17.000000 18.000000 19.000000 25.000000 26.000000 27.000000 28.000000 ...
[TC_L1_02] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_02 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_03
[TC_L1_03] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 11.000000 ...
[TC_L1_03] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 11.000000 ...
[TC_L1_03] PASSED (exact match, 0/45 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_03 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_04
[TC_L1_04] Output: 1.000000 10.000000 11.000000 19.000000 20.000000 21.000000 28.000000 29.000000 30.000000 31.000000 ...
[TC_L1_04] Golden: 1.000000 10.000000 11.000000 19.000000 20.000000 21.000000 28.000000 29.000000 30.000000 31.000000 ...
[TC_L1_04] PASSED (exact match, 0/45 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_04 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_05
[TC_L1_05] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_05] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_05] PASSED (exact match, 0/2080 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_05 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_06
[TC_L1_06] Output: 1.000000 65.000000 66.000000 129.000000 130.000000 131.000000 193.000000 194.000000 195.000000 196.000000 ...
[TC_L1_06] Golden: 1.000000 65.000000 66.000000 129.000000 130.000000 131.000000 193.000000 194.000000 195.000000 196.000000 ...
[TC_L1_06] PASSED (exact match, 0/2080 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_06 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_07
[TC_L1_07] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_07] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_07] PASSED (exact match, 0/5050 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_07 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_08
[TC_L1_08] Output: 1.000000 101.000000 102.000000 201.000000 202.000000 203.000000 301.000000 302.000000 303.000000 304.000000 ...
[TC_L1_08] Golden: 1.000000 101.000000 102.000000 201.000000 202.000000 203.000000 301.000000 302.000000 303.000000 304.000000 ...
[TC_L1_08] PASSED (exact match, 0/5050 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_08 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_09
[TC_L1_09] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_09] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_09] PASSED (exact match, 0/32896 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_09 (4 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_10
[TC_L1_10] Output: 1.000000 257.000000 258.000000 513.000000 514.000000 515.000000 769.000000 770.000000 771.000000 772.000000 ...
[TC_L1_10] Golden: 1.000000 257.000000 258.000000 513.000000 514.000000 515.000000 769.000000 770.000000 771.000000 772.000000 ...
[TC_L1_10] PASSED (exact match, 0/32896 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_10 (4 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_11
[TC_L1_11] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_11] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_11] PASSED (exact match, 0/125250 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_11 (16 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_12
[TC_L1_12] Output: 1.000000 501.000000 502.000000 1001.000000 1002.000000 1003.000000 1501.000000 1502.000000 1503.000000 1504.000000 ...
[TC_L1_12] Golden: 1.000000 501.000000 502.000000 1001.000000 1002.000000 1003.000000 1501.000000 1502.000000 1503.000000 1504.000000 ...
[TC_L1_12] PASSED (exact match, 0/125250 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_12 (16 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_13
[TC_L1_13] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_13] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_13] PASSED (exact match, 0/524800 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_13 (75 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_14
[TC_L1_14] Output: 1.000000 1025.000000 1026.000000 2049.000000 2050.000000 2051.000000 3073.000000 3074.000000 3075.000000 3076.000000 ...
[TC_L1_14] Golden: 1.000000 1025.000000 1026.000000 2049.000000 2050.000000 2051.000000 3073.000000 3074.000000 3075.000000 3076.000000 ...
[TC_L1_14] PASSED (exact match, 0/524800 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_14 (70 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_15
[TC_L1_15] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 14.000000 15.000000 ...
[TC_L1_15] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 14.000000 15.000000 ...
[TC_L1_15] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_15 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_16
[TC_L1_16] Output: 1.000000 13.000000 14.000000 25.000000 26.000000 27.000000 37.000000 38.000000 39.000000 40.000000 ...
[TC_L1_16] Golden: 1.000000 13.000000 14.000000 25.000000 26.000000 27.000000 37.000000 38.000000 39.000000 40.000000 ...
[TC_L1_16] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_16 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_17
[TC_L1_17] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_17] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_17] PASSED (exact match, 0/136 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_17 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_18
[TC_L1_18] Output: 1.000000 33.000000 34.000000 65.000000 66.000000 67.000000 97.000000 98.000000 99.000000 100.000000 ...
[TC_L1_18] Golden: 1.000000 33.000000 34.000000 65.000000 66.000000 67.000000 97.000000 98.000000 99.000000 100.000000 ...
[TC_L1_18] PASSED (exact match, 0/136 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_18 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_19
[TC_L1_19] Output: 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 ...
[TC_L1_19] Golden: 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 ...
[TC_L1_19] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_19 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_20
[TC_L1_20] Output: 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 ...
[TC_L1_20] Golden: 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 ...
[TC_L1_20] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_20 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_21
[TC_L1_21] Output: 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 ...
[TC_L1_21] Golden: 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 ...
[TC_L1_21] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_21 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_22
[TC_L1_22] Output: 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 ...
[TC_L1_22] Golden: 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 10000000000.000000 ...
[TC_L1_22] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_22 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_23
[TC_L1_23] Output: -1.000000 -2.000000 -3.000000 -4.000000 -5.000000 -6.000000 -7.000000 -8.000000 -10.000000 -11.000000 ...
[TC_L1_23] Golden: -1.000000 -2.000000 -3.000000 -4.000000 -5.000000 -6.000000 -7.000000 -8.000000 -10.000000 -11.000000 ...
[TC_L1_23] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_23 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_24
[TC_L1_24] Output: -1.000000 -9.000000 -10.000000 -17.000000 -18.000000 -19.000000 -25.000000 -26.000000 -27.000000 -28.000000 ...
[TC_L1_24] Golden: -1.000000 -9.000000 -10.000000 -17.000000 -18.000000 -19.000000 -25.000000 -26.000000 -27.000000 -28.000000 ...
[TC_L1_24] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_24 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_25
[TC_L1_25] Output: inf inf inf inf inf inf inf inf inf inf ...
[TC_L1_25] Golden: inf inf inf inf inf inf inf inf inf inf ...
[TC_L1_25] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_25 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_26
[TC_L1_26] Output: inf inf inf inf inf inf inf inf inf inf ...
[TC_L1_26] Golden: inf inf inf inf inf inf inf inf inf inf ...
[TC_L1_26] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_26 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_27
[TC_L1_27] Output: nan nan nan nan nan nan nan nan nan nan ...
[TC_L1_27] Golden: nan nan nan nan nan nan nan nan nan nan ...
[TC_L1_27] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_27 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_28
[TC_L1_28] Output: nan nan nan nan nan nan nan nan nan nan ...
[TC_L1_28] Golden: nan nan nan nan nan nan nan nan nan nan ...
[TC_L1_28] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_28 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_29
[TC_L1_29] Output: 1.000000 0.000000 -1.000000 340282346638528859811704183484516925440.000000 0.000000 -340282346638528859811704183484516925440.000000 0.000000 1.000000 -1.000000 340282346638528859811704183484516925440.000000 ...
[TC_L1_29] Golden: 1.000000 0.000000 -1.000000 340282346638528859811704183484516925440.000000 0.000000 -340282346638528859811704183484516925440.000000 0.000000 1.000000 -1.000000 340282346638528859811704183484516925440.000000 ...
[TC_L1_29] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_29 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_30
[TC_L1_30] Output: 1.000000 0.000000 -1.000000 -1.000000 340282346638528859811704183484516925440.000000 0.000000 340282346638528859811704183484516925440.000000 0.000000 -340282346638528859811704183484516925440.000000 0.000000 ...
[TC_L1_30] Golden: 1.000000 0.000000 -1.000000 -1.000000 340282346638528859811704183484516925440.000000 0.000000 340282346638528859811704183484516925440.000000 0.000000 -340282346638528859811704183484516925440.000000 0.000000 ...
[TC_L1_30] PASSED (exact match, 0/36 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_30 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_31
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_31 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_32
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_32 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_34
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_34 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_35
[TC_L1_35] Output: 
[TC_L1_35] Golden: 
[TC_L1_35] PASSED (exact match, 0/0 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_35 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_36
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_36 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_37
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_37 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_38
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_38 (0 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_41
[TC_L1_41] Output: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_41] Golden: 1.000000 2.000000 3.000000 4.000000 5.000000 6.000000 7.000000 8.000000 9.000000 10.000000 ...
[TC_L1_41] PASSED (exact match, 0/52433920 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_41 (7632 ms)
[ RUN      ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_42
[TC_L1_42] Output: 1.000000 10241.000000 10242.000000 20481.000000 20482.000000 20483.000000 30721.000000 30722.000000 30723.000000 30724.000000 ...
[TC_L1_42] Golden: 1.000000 10241.000000 10242.000000 20481.000000 20482.000000 20483.000000 30721.000000 30722.000000 30723.000000 30724.000000 ...
[TC_L1_42] PASSED (exact match, 0/52433920 mismatches)
[       OK ] Strttp/StrttpArch35Test.CsvDriven/TC_L1_42 (7505 ms)
[----------] 55 tests from Strttp/StrttpArch35Test (15375 ms total)

[----------] Global test environment tear-down
[==========] 56 tests from 2 test suites ran. (22783 ms total)
[  PASSED  ] 56 tests.
[PASS] strttp_test

========================================
Test Summary:
  Passed:  40 - sasum blasLtMatmul blasLtMatrixTransform scopy sdot sgbmv sgels_batched gemm_grouped_batched gemm_grouped_batched_ex sgemv gemv_batched sgeqrf_batched sger sgetrf_batched sgetri_batched isamax matinv_batched snrm2 srotg srotm srotmg saxpy ssbmv sscal scalex sspmv stbsv sswap ssymm ssymv ssyr ssyr2 stbmv stpmv stpsv stpttr strmm strmv strsv strttp
  Skipped: 14 - caxpy colwise_mul complex_mat_dot cdot cgemv cgerc getrs_batched iamax csrot cscal spr spr2 cswap ctrmv (not supported on ascend950)
  Failed:  3 - gemm_ex gemm_batched_ex isamin
========================================