已关闭
[Bug]: ATK 运行 ATB调用 950 FIAv5 PA 报错507015 #1
ivanS创建于 6月5日关闭于 6月8日
6月5日 添加了label：bug
ivanS
6月5日 评论：
6月5日 评论：
atk日志
[2026-06-05 16:55:09,206] [INFO] [MainProcess] [79874] [base.py:148] task_output_path: /home/d60039526/ATK_CIDA/atb/infer/SelfAttentionOperation1/atk_output/all_SelfAttentionOperation_MLA_Model_2026-06-06-00-55-09-093223
[2026-06-05 16:55:09,206] [INFO] [MainProcess] [79874] [base.py:119] start atk task: namespace(case='all_SelfAttentionOperation_MLA_Model.json', plugin_path='execute_self_attention_mla.py', atb_api_path='../../common/', nodes='nodes_SelfAttentionOperation_MLA_Perf.yaml', max_task=1, single_process=True, white_list='[2]', fast_case=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, mssanitizer=False, ms_log_path=None, trace=False, timeout=None, db_timeout=None, db_path=None, task=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, need_time=True, ct_tool=False, nodes_list={})
[2026-06-05 16:55:09,237] [INFO] [MainProcess] [79874] [base.py:168] start single process run
[2026-06-05 16:55:09,238] [WARNING] [MainProcess] [79874] [atb_base_api.py:216] ATK NPUBackend custom_data None-guard skipped: type object 'NPUBackend' has no attribute 'custom_data_computation'
[2026-06-05 16:55:09,240] [INFO] [MainProcess] [79874] [registry.py:61] execute_atb_self_attention_mla is already registered, repeated registration will coverage it, new module path is execute_self_attention_mla
[2026-06-05 16:55:09,241] [INFO] [MainProcess] [79874] [single_process_main.py:207] report api name: all_SelfAttentionOperation_MLA_Model
[2026-06-05 16:55:09,242] [INFO] [MainProcess] [79874] [opp_tasks.py:131] [case 2] start CREATE DATASET.
[2026-06-05 16:55:09,245] [INFO] [MainProcess] [79874] [opp_tasks.py:153] [case 2] SUCCESS CREATE DATASET.
[2026-06-05 16:55:09,245] [INFO] [MainProcess] [79874] [opp_tasks.py:264] [case 2] start run ATB OPP TASK.
[2026-06-05 16:55:09,251] [INFO] [MainProcess] [79874] [atb_base_api.py:51] AtbBaseFunctionApi init
[2026-06-05 16:55:33,334] [INFO] [MainProcess] [79874] [atb_base_api.py:51] AtbBaseFunctionApi init
[2026-06-05 16:55:33,621] [INFO] [MainProcess] [79874] [patch_getenv.py:15] get env ASCEND_RT_VISIBLE_DEVICES = 1
[2026-06-05 16:55:33,628] [INFO] [MainProcess] [79874] [opp_executor.py:301] [case 2][RUN OPP TASK] START Init by input data.
[2026-06-05 16:55:33,629] [INFO] [MainProcess] [79874] [execute_self_attention_mla.py:177] MLA executor init: maskType=0, headNum=2, kvHeadNum=0, seqLen=[16], ntokens=16, n_tensors=4
[2026-06-05 16:55:33,629] [INFO] [MainProcess] [79874] [opp_executor.py:303] [case 2][RUN OPP TASK] SUCCESS Init by input data.
[2026-06-05 16:55:33,629] [INFO] [MainProcess] [79874] [opp_executor.py:60] [case 2][RUN OPP TASK] Start Execute OPP on ['performance_device']
[2026-06-05 16:55:33,629] [INFO] [MainProcess] [79874] [execute_self_attention_mla.py:198] MLA atb_input: n_tensors=4, in_formats=[2, 2, 2, 2]
[2026-06-05 16:55:33,670] [INFO] [MainProcess] [79874] [atb_base_api.py:98] [atb_input_function] before create_variant_pack: run_params_len=16 run_params='{"seqLen": [16]}'
[2026-06-05 16:55:34,536] [INFO] [MainProcess] [79874] [backend.py:63] =完成 warmup 阶段==
[2026-06-05 16:56:07,988] [ERROR] [MainProcess] [79874] [opp_tasks.py:227] case 2 run opp failed:
error code '507015' not found in error_dict.
[2026-06-05 16:56:07,989] [ERROR] [MainProcess] [79874] [opp_tasks.py:228]
error code '507015' not found in error_dict.
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 222, in run_opp_case
ret, output_info_list = opp_executor.run_opp_case()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/common/utils.py", line 82, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 78, in run_opp_case
perf_output_info = self._execute_performance_tasks(perf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 510, in _execute_performance_tasks
self.run_device_performance(perf)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/common/utils.py", line 82, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 149, in run_device_performance
self.backend.run_device_perf(perf)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 230, in run_device_perf
self.synchronize()
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 169, in synchronize
self.check_error_code(error_code)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 74, in check_error_code
raise ATKRuntimeError(f"\nerror code '{error_code}' not found in error_dict.")
atk.tasks.backends.backend.ATKRuntimeError:
error code '507015' not found in error_dict.
[2026-06-05 16:56:07,990] [ERROR] [MainProcess] [79874] [opp_tasks.py:282] Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 222, in run_opp_case
ret, output_info_list = opp_executor.run_opp_case()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/common/utils.py", line 82, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 78, in run_opp_case
perf_output_info = self._execute_performance_tasks(perf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 510, in _execute_performance_tasks
self.run_device_performance(perf)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/common/utils.py", line 82, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 149, in run_device_performance
self.backend.run_device_perf(perf)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 230, in run_device_perf
self.synchronize()
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 169, in synchronize
self.check_error_code(error_code)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 74, in check_error_code
raise ATKRuntimeError(f"\nerror code '{error_code}' not found in error_dict.")
atk.tasks.backends.backend.ATKRuntimeError:
error code '507015' not found in error_dict.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 271, in run_opp_task
run_opp_case(dataset_executor, task_result)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 229, in run_opp_case
raise ATKRuntimeError("case %s run opp case failed: %s", task_result.case_config.id) from e
atk.tasks.backends.backend.ATKRuntimeError: ('case %s run opp case failed: %s', 2)
[2026-06-05 16:56:07,990] [ERROR] [MainProcess] [79874] [opp_tasks.py:283] case 2 run opp failed: ('case %s run opp case failed: %s', 2)
[2026-06-05 16:56:07,990] [ERROR] [MainProcess] [79874] [single_process_main.py:185] run_opp_task fail, node_name:0, error: run opp task failed!
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 222, in run_opp_case
ret, output_info_list = opp_executor.run_opp_case()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/common/utils.py", line 82, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 78, in run_opp_case
perf_output_info = self._execute_performance_tasks(perf)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 510, in _execute_performance_tasks
self.run_device_performance(perf)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/common/utils.py", line 82, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 149, in run_device_performance
self.backend.run_device_perf(perf)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 230, in run_device_perf
self.synchronize()
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 169, in synchronize
self.check_error_code(error_code)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/backends/atb_backend.py", line 74, in check_error_code
raise ATKRuntimeError(f"\nerror code '{error_code}' not found in error_dict.")
atk.tasks.backends.backend.ATKRuntimeError:
error code '507015' not found in error_dict.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 271, in run_opp_task
run_opp_case(dataset_executor, task_result)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 229, in run_opp_case
raise ATKRuntimeError("case %s run opp case failed: %s", task_result.case_config.id) from e
atk.tasks.backends.backend.ATKRuntimeError: ('case %s run opp case failed: %s', 2)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/single_process_main.py", line 181, in run_node
res = run_opp_task(task_result.model_dump())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 284, in run_opp_task
raise RuntimeError("run opp task failed!") from run_exec
RuntimeError: run opp task failed!
[2026-06-05 16:56:07,991] [INFO] [MainProcess] [79874] [opp_tasks.py:264] [case 2] start run ATB OPP TASK.
[2026-06-05 16:56:07,991] [INFO] [MainProcess] [79874] [atb_base_api.py:51] AtbBaseFunctionApi init
[2026-06-05 16:56:07,998] [INFO] [MainProcess] [79874] [atb_base_api.py:51] AtbBaseFunctionApi init
[2026-06-05 16:56:08,000] [ERROR] [MainProcess] [79874] [opp_tasks.py:282] Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 271, in run_opp_task
run_opp_case(dataset_executor, task_result)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 219, in run_opp_case
opp_executor.before_run(data)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 300, in before_run
input_dataset.to_device(self.backend.get_device())
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 188, in to_device
self.kwargs = self._set_input_device(self.kwargs, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 266, in _set_input_device
return self._set_input_data_by_dict(data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 253, in _set_input_data_by_dict
input_data[name] = self._set_input_device(data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 268, in _set_input_device
return _set_input_data_by_tensor(data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 261, in _set_input_data_by_tensor
return input_data.to(device)
^^^^^^^^^^^^^^^^^^^^^
RuntimeError: ACL stream synchronize failed, error code:507015
[2026-06-05 16:56:08,000] [ERROR] [MainProcess] [79874] [opp_tasks.py:283] case 2 run opp failed: ACL stream synchronize failed, error code:507015
[2026-06-05 16:56:08,000] [ERROR] [MainProcess] [79874] [single_process_main.py:185] run_opp_task fail, node_name:1, error: run opp task failed!
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 271, in run_opp_task
run_opp_case(dataset_executor, task_result)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 219, in run_opp_case
opp_executor.before_run(data)
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/executors/opp_executor.py", line 300, in before_run
input_dataset.to_device(self.backend.get_device())
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 188, in to_device
self.kwargs = self._set_input_device(self.kwargs, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 266, in _set_input_device
return self._set_input_data_by_dict(data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 253, in _set_input_data_by_dict
input_data[name] = self._set_input_device(data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 268, in _set_input_device
return _set_input_data_by_tensor(data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/configs/dataset_config.py", line 261, in _set_input_data_by_tensor
return input_data.to(device)
^^^^^^^^^^^^^^^^^^^^^
RuntimeError: ACL stream synchronize failed, error code:507015
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/single_process_main.py", line 181, in run_node
res = run_opp_task(task_result.model_dump())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.13/lib/python3.11/site-packages/atk/tasks/opp_tasks.py", line 284, in run_opp_task
raise RuntimeError("run opp task failed!") from run_exec
RuntimeError: run opp task failed!
[2026-06-05 16:56:08,001] [ERROR] [MainProcess] [79874] [csv_report.py:424] All cases is running failed.
XXiaoyue
6月5日 评论：
6月5日 评论：
507015 --- AI Core执行异常，非工具问题。
Wiki中已有参考FAQ，用户请先查阅：https://wiki.huawei.com/domains/143914/wiki/352630/WIKI202509018099576
- 查看plog日志（/root/ascend/log/debug/plog），找对应算子开发或CANN定位解决
- 主要检查输入构造是否正确，大概率是kernel执行失败
6月8日 添加了label：resolved
⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：
在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用
<TOKEN>等占位符替代原有内容。环境信息
🐛 问题描述
atk task -n node_perf.yaml -c result/ATB_PagedAttention_gen/json/PagedAttention_Model.json -p ./ --task performance_device -ap xxx/common -mt 10
后报错507015
case:
日志：
print screen 日志：
欢迎加入社区，感谢您对社区的贡献 🎉!