Star60
77
代码
代码
Issues176
Pull Requests50
项目讨论
Wiki
分析
Star60
77
已关闭
[Bug]: Qwen3.5-0.8B中qwen3_5_llm_decode.onnx模型在300iduo上转换报错和README.md部分资料描述不清晰 #449
张朝晖创建于  27 天前关闭于  20 天前
张朝晖
张朝晖成员
27 天前 创建

Checklist

🐞 问题详细描述

1、按照README.md在300iduo上执行转换qwen3_5_llm_decode.onnx时报错。
转换命令：
export TUNE_BANK_PATH=...
$Convert --fmk=ONNX
--modelFile=qwen3_5_onnx_static/decoder/qwen3_5_llm_decode.onnx
--outputFile=qwen3_5_onnx_static/qwen3_5_llm_decode
--optimize=ascend_oriented
--configFile=configs/config_decoder.ini
--saveType=MINDIR
报错：

[WARNING] DEVICE(2098843,ffff9e884020,converter_lite):2026-08-29-17:04:38.826.707 [ugin/ascend/res_manager/op_adapter/op_adapter_map.cc:83] FindAdapter] Can't find OpAdapter for Scatter
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1787994283.038530 2098843 message_lite.cc:411] ge.proto.ModelDef exceeded maximum protobuf size of 2GB: 4034472498
./home/miniconda3/envs/ci_312/lib/python3.12/site-packages/torch/cuda/__init__.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:54.546.828 [mindspore-lite/tools/converter/adapter/acl/cxx_api_lite/cxx_api/model/acl/model_converter.cc:286] BuildAirModel] Call aclgrphBuildModel fail: [PID: 2098843] 2026-08-29-17:05:45.803.406 Inner_Error_Compile_Fail(E90003): Compile operator failed, cause: Template constraint, detailed information: Tiling func of op_type IncreFlashAttention failed, failure details:
Compile_info: {'device_id': None, '_cube_vector_core_type': 'AiCore'}
Inputs: {'shape': (1, 8, 1, 256), 'ori_shape': (1, 8, 1, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 8, 1, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 8, 'range': [[1, 1], [8, 8], [1, 1], [256, 256]], 'param_name': 'query_in__'}
({'shape': (1, 2, 2048, 256), 'ori_shape': (1, 2, 2048, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 2, 2048, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 2, 'range': [[1, 1], [2, 2], [2048, 2048], [256, 256]], 'param_name': 'key_in__'},)
({'shape': (1, 2, 2048, 256), 'ori_shape': (1, 2, 2048, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 2, 2048, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 2, 'range': [[1, 1], [2, 2], [2048, 2048], [256, 256]], 'param_name': 'value_in__'},)
None
{'shape': (1, 1, 1, 2048), 'ori_shape': (1, 1, 1, 2048), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'bool', 'addr_type': 0, 'total_shape': [1, 1, 1, 2048], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 1, 'range': [[1, 1], [1, 1], [1, 1], [2048, 2048]], 'param_name': 'atten_mask_in__'}
None
None
None
None
None
None
None
None
None
None
Outputs: {'shape': (1, 8, 1, 256), 'ori_shape': (1, 8, 1, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 8, 1, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 8, 'range': [[1, 1], [8, 8], [1, 1], [256, 256]], 'param_name': 'attention_out_out_'}
[OP_TILING] Attrs: [{'name': 'num_heads', 'dtype': 'int', 'value': 8}, {'name': 'scale_value', 'dtype': 'float', 'value': 0.0625}, {'name': 'input_layout', 'dtype': 'str', 'value': 'BNSD'}, {'name': 'num_key_value_heads', 'dtype': 'int', 'value': 2}, {'name': 'block_size', 'dtype': 'int', 'value': 0}, {'name': 'inner_precise', 'dtype': 'int', 'value': 1}, {'name': 'ascendc_op_para_size', 'dtype': 'int', 'value': 2097152}].
TraceBack (most recent call last):
        On 300I DUO, when gqa > 1 or qlen > 1, it requires that gqa * qlen <= 32 and headDim <=128.[FUNC:CheckInputFormatAndLimits][FILE:incre_flash_attention_tiling_check.cpp][LINE:372][FUNC:TeInnerErrMessageReport][FILE:common_utils.cc][LINE:397]
        Op IncreFlashAttention tiling failed[FUNC:TbeOptilingPyInterfaceNew][FILE:op_tiling_py.cc][LINE:1962][FUNC:TeInnerErrMessageReport][FILE:common_utils.cc][LINE:397]
        TbeOpTilingPyInterfaceOld failed.[FUNC:DoOpTilingForCompile][FILE:op_tiling_py.cc][LINE:2060][FUNC:TeInnerErrMessageReport][FILE:common_utils.cc][LINE:397]
        Compile operator failed, cause: Template constraint, detailed information: Tiling func of op_type IncreFlashAttention failed, failure details:
Compile_info: {'device_id': None, '_cube_vector_core_type': 'AiCore'}
Inputs: {'shape': (1, 8, 1, 256), 'ori_shape': (1, 8, 1, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 8, 1, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 8, 'range': [[1, 1], [8, 8], [1, 1], [256, 256]], 'param_name': 'query_in__'}
({'shape': (1, 2, 2048, 256), 'ori_shape': (1, 2, 2048, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 2, 2048, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 2, 'range': [[1, 1], [2, 2], [2048, 2048], [256, 256]], 'param_name': 'key_in__'},)
({'shape': (1, 2, 2048, 256), 'ori_shape': (1, 2, 2048, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 2, 2048, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 2, 'range': [[1, 1], [2, 2], [2048, 2048], [256, 256]], 'param_name': 'value_in__'},)
None
{'shape': (1, 1, 1, 2048), 'ori_shape': (1, 1, 1, 2048), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'bool', 'addr_type': 0, 'total_shape': [1, 1, 1, 2048], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 1, 'range': [[1, 1], [1, 1], [1, 1], [2048, 2048]], 'param_name': 'atten_mask_in__'}
None
None
None
None
None
None
None
None
None
None
Outputs: {'shape': (1, 8, 1, 256), 'ori_shape': (1, 8, 1, 256), 'format': 'NCHW', 'sub_format': 0, 'ori_format': 'NCHW', 'dtype': 'float16', 'addr_type': 0, 'total_shape': [1, 8, 1, 256], 'slice_offset': (), 'L1_addr_offset': 0, 'L1_fusion_type': -1, 'L1_workspace_size': -1, 'valid_shape': (), 'split_index': 0, 'atomic_type': '', 'input_c_values': 8, 'range': [[1, 1], [8, 8], [1, 1], [256, 256]], 'param_name': 'attention_out_out_'}
[OP_TILING] Attrs: [{'name': 'num_heads', 'dtype': 'int', 'value': 8}, {'name': 'scale_value', 'dtype': 'float', 'value': 0.0625}, {'name': 'input_layout', 'dtype': 'str', 'value': 'BNSD'}, {'name': 'num_key_value_heads', 'dtype': 'int', 'value': 2}, {'name': 'block_size', 'dtype': 'int', 'value': 0}, {'name': 'inner_precise', 'dtype': 'int', 'value': 1}, {'name': 'ascendc_op_para_size', 'dtype': 'int', 'value': 2097152}].
        Failed to compile Op /Custom_2. oppath is /usr/local/Ascend/cann-9.1.0/opp/built-in/op_impl/ai_core/tbe/impl/ops_transformer/dynamic/incre_flash_attention.py Compile failed with errormsg/stack:  and optype is IncreFlashAttention.
        Compile op[/Custom_2] failed, oppath[/usr/local/Ascend/cann-9.1.0/opp/built-in/op_impl/ai_core/tbe/impl/ops_transformer/dynamic/incre_flash_attention.py], optype[IncreFlashAttention], taskID[1802]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:369]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281470961543232] failed to recompile single op[/Custom_2][FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:1169]
        [SubGraphOpt][Compile][ParalCompOp] Thread[281470961543232] failed when processing the task that had failed to compile.[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1216]
        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1175]
        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1452]
        Call OptimizeFusedGraph failed, ret:4294967295, engine_name:AIcoreEngine, graph_name:partition0_rank353_new_sub_graph737[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:124]
        subgraph 352 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:1045]
        Assert ((DoSubgraphPartitionWithMode(graph_node, compute_graph, session_id, EnginePartitioner::Mode::kAtomicEnginePartitioning, "AtomicEngine")) == ge::SUCCESS) failed[FUNC:OptimizeSubgraph][FILE:graph_manager.cc][LINE:4019]
        build graph failed, graph id:0, ret:1343225857[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1629]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:6081]
        The argument is invalid. Reason: rtGetDevMsg execution failed, the context is a null pointer.

[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:56.874.497 [mindspore-lite/tools/converter/adapter/acl/cxx_api_lite/cxx_api/model/acl/model_converter.cc:379] LoadMindIR] Convert model from MindIR to OM failed
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.572 [mindspore-lite/tools/converter/adapter/acl/src/acl_pass_impl.cc:986] ConvertGraphToOm] Model converter load mindir failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.640 [mindspore-lite/tools/converter/adapter/acl/src/acl_pass_impl.cc:1036] BuildGraph] Convert graph  to om failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.656 [mindspore-lite/tools/converter/adapter/acl/src/acl_pass_impl.cc:1441] Run] Build graph failed!
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.676 [mindspore-lite/tools/converter/adapter/acl/acl_pass.cc:42] Run] Acl pass impl run failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.692 [mindspore-lite/tools/converter/anf_transform.cc:576] RunConvertPass] Acl pass failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.726 [mindspore-lite/tools/converter/anf_transform.cc:775] RunPass] Run convert pass failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.176.739 [mindspore-lite/tools/converter/anf_transform.cc:876] TransformFuncGraph] Proc online transform failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.184.179 [mindspore-lite/tools/converter/anf_transform.cc:988] Transform] optimizer failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.184.213 [mindspore-lite/tools/converter/converter_funcgraph.cc:655] Optimize] Transform anf graph failed.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.184.249 [mindspore-lite/tools/converter/converter.cc:1190] HandleGraphCommon] Optimize func graph failed: -2 NULL pointer returned.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.184.270 [mindspore-lite/tools/converter/converter.cc:1139] Convert] Handle graph failed: -2 NULL pointer returned.
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.184.284 [mindspore-lite/tools/converter/converter.cc:1332] RunConverter] Convert model failed
[ERROR] LITE(2098843,ffff9e884020,converter_lite):2026-08-29-17:05:57.184.314 [mindspore-lite/tools/converter/cxx_api/converter.cc:317] Convert] Convert model failed, ret=NULL pointer returned.
ERROR [mindspore-lite/tools/converter/converter_lite/main.cc:107] main] Convert failed. Ret: NULL pointer returned.
Convert failed. Ret: NULL pointer returned.

2、模型导出前建议加一步获取模型权重与源码
image.png

3、mindspore-lite/examples/base_models/qwen3_5_0.8b/configs/config_prefill.ini，‘ND’应该为“ND”
image.png

4、推理示例输出部分实际生成描述：Generated Response: This is a surreal, abstract digital artwork featuring two bananas — one slightly behind and to the left of the other — rendered in a vibrant, multi-colored, almost liquid-like material. The bananas are not realistic but instead appear to be made of a glossy, translucent substance that reflects and refracts light, giving them a shimmering, almost melting appearance.

The background consists of horizontal stripes in soft pastel tones: pink, lavender, mint green, and pale gray — creating a dreamy, striped pattern that contrasts with the colorful subject.

The bananas cast a soft, diffused shadow beneath them, suggesting depth and dimensionality. The overall
image.png

5、qwen3_5_llm_decode.onnx转换命令前确定是否需要设置export TUNE_BANK_PATH=...
image.png

详细的环境信息描述

300iduo

其他辅助信息

版本信息

mindspore_lite:master_20260828

Thanks for contributing 🎉!

likedislike
张朝晖张朝晖成员
27 天前 添加了label：bug
此处折叠了7条事件消息 查看更多
张朝晖张朝晖成员
27 天前 issue优先级由 次要 改变为 严重
YeFeng_24
YeFeng_24成员
27 天前 评论：
  1. 910B功能，精度，性能均ok；300IDUO decode转换存在问题；300IDUO所有模型的推理功能与精度以及性能暂时未验证（decode模型转换失败导致验证堵塞）
  2. ND单引号和双引号导致功能存在问题；（910B和300IDUO都会失败）；
  3. 精度问题；
likedislike
Xxiongpan成员
26 天前 关联了pull request：修正qwen3.5-0.8b资料
X
xiongpan成员
25 天前 评论：
likedislike
Xxiongpan成员
25 天前 移除了负责人 YeFeng_24
此处折叠了5条事件消息 查看更多
Xxiongpan成员
25 天前 关联组件 由 [B-SIG-InnerDelivery] 改变为 [B-ComponentTest]
张朝晖
张朝晖成员
20 天前 评论：

1、Version：
lite-boost：https://repo.mindspore.cn/mindspore/mindspore-lite/newest/lite/release/linux/aarch64/cloud_fusion/python312/
cann：cann_9.2.0_20260825

2、Report:image.png

3、Conclusion： pass

likedislike
张朝晖张朝晖成员
20 天前 issue状态由 VALIDATION 改变为 DONE
张朝晖张朝晖成员
20 天前 关闭了 issue