[Usage]: develop_examples_and_tools分支中gr模型样例运行失败-RecSDK-AtomGit

已关闭

[Usage]: develop_examples_and_tools分支中gr模型样例运行失败 #1109

Sinon创建于 4月25日关闭于 5月26日

4月25日创建

环境信息

Debian 12 x86_64
MindSDK 26.0.0.B090

使用场景及问题

报错信息如下：

Traceback (most recent call last):
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/pretrain_gr_ranking.py", line 37, in <module>
    from model import get_ranking_model
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/model/__init__.py", line 2, in <module>
    from model.ranking_gr import RankingGR
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/model/ranking_gr.py", line 25, in <module>
    from modules.hstu_block import HSTUBlock
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/modules/hstu_block.py", line 12, in <module>
    from modules.fused_hstu_layer import FusedHSTULayer
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/modules/fused_hstu_layer.py", line 23, in <module>
    from ops.fused_hstu_op import fused_hstu_op
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/ops/fused_hstu_op.py", line 34, in <module>
    from ops.triton_ops.triton_hstu_attention import (
  File "/home/RecSDK/TorchRecV2/gr/recsys-examples/examples/hstu/ops/triton_ops/triton_hstu_attention.py", line 535, in <module>
    @triton.autotune(
     ^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/runtime/autotuner.py", line 368, in decorator
    return Autotuner(fn, fn.arg_names, configs, key, reset_to_zero, restore_value, pre_hook=pre_hook,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/runtime/autotuner.py", line 130, in __init__
    self.do_bench = driver.active.get_benchmarker()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/runtime/driver.py", line 23, in __getattr__
    self._initialize_obj()
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/runtime/driver.py", line 20, in _initialize_obj
    self._obj = self._init_fn()
                ^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/runtime/driver.py", line 9, in _create_driver
    return actives[0]()
           ^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend/driver.py", line 167, in __init__
    self.utils = NPUUtils()
                 ^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend/driver.py", line 66, in __init__
    so = _exec_cmd_with_lock(tmp_src_path, "npu_utils", 
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend/driver.py", line 238, in _exec_cmd_with_lock
    return fn()
           ^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend/driver.py", line 67, in <lambda>
    lambda: self._write_npu_utils_and_compile(tmp_src_path, src))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend/driver.py", line 82, in _write_npu_utils_and_compile
    return _build_npu_ext("npu_utils", None, tmp_src_path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend/utils.py", line 417, in _build_npu_ext
    raise RuntimeError(f"Failed to compile {src_path}, error: {result.stderr},cmd={cc_cmd}")
RuntimeError: Failed to compile /tmp/tmpyjqmuv91/npu_utils.cpp, error: /tmp/tmpyjqmuv91/npu_utils.cpp:324:44: error: ‘RT_LIMIT_TYPE_SIMT_WARP_STACK_SIZE’ is not a member of ‘rtLimitType_t’ {aka ‘tagRtLimitType’}
  324 |         {"WARP_STACK_SIZE", rtLimitType_t::RT_LIMIT_TYPE_SIMT_WARP_STACK_SIZE},
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/tmpyjqmuv91/npu_utils.cpp:327:3: error: could not convert ‘{{"LOW_POWER_TIMEOUT", RT_LIMIT_TYPE_LOW_POWER_TIMEOUT}, {"WARP_STACK_SIZE", <expression error>}, {"DVG_WARP_STACK_SIZE", RT_LIMIT_TYPE_SIMT_DVG_WARP_STACK_SIZE}, {"STACK_SIZE", RT_LIMIT_TYPE_STACK_SIZE}}’ from ‘<brace-enclosed initializer list>’ to ‘const std::unordered_map<std::__cxx11::basic_string<char>, tagRtLimitType>’
  327 |   };
      |   ^
      |   |
      |   <brace-enclosed initializer list>
,cmd=['/usr/local/gcc11.2.0/bin/g++', '/tmp/tmpyjqmuv91/npu_utils.cpp', '-w', '-I/usr/local/python3.11.0/include/python3.11', '-I/usr/local/python3.11.0/lib/python3.11/site-packages/triton/backends/ascend', '-I/usr/local/Ascend/cann-9.0.0/pkg_inc', '-I/usr/local/Ascend/cann-9.0.0/pkg_inc/profiling', '-I/usr/local/Ascend/cann-9.0.0/include', '-I/usr/local/Ascend/cann-9.0.0/include/experiment', '-I/usr/local/Ascend/cann-9.0.0/include/experiment/msprof', '-I/usr/local/python3.11.0/lib/python3.11/site-packages/pybind11/include', '-L/usr/local/Ascend/cann-9.0.0/lib64', '-lruntime', '-lascendcl', '-I/usr/local/python3.11.0/lib/python3.11/site-packages/torch/include', '-I/usr/local/python3.11.0/lib/python3.11/site-packages/torch_npu/include', '-D_GLIBCXX_USE_CXX11_ABI=1', '-L/usr/local/python3.11.0/lib/python3.11/site-packages/torch_npu/lib', '-ltorch_npu', '-std=c++17', '-shared', '-fPIC', '-Winvalid-pch', '-o', '/tmp/tmpyjqmuv91/npu_utils.cpython-311-x86_64-linux-gnu.so']
[ERROR] 2026-04-25-16:28:39 (PID:4918, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
E0425 16:28:40.862000 4847 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 0 (pid: 4918) of binary: /usr/local/python3.11.0/bin/python3.11
Traceback (most recent call last):
  File "/usr/local/python3.11.0/bin/torchrun", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in main
    run(args)
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
    elastic_launch(
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.0/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
pretrain_gr_ranking.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-04-25_16:28:40
  host      : localhost.localdomain
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 4918)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
[ERROR] 2026-04-25-16:28:40 (PID:4847, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

ascend-robot成员

4月25日添加了label：usage

Sinon

4月25日修改了issue 的描述

Sinon

4月25日修改了issue 的描述

taoqiuyu1998成员

4月25日评论：

/label add triaged

ascend-robot成员

4月25日添加了label：triaged

taoqiuyu1998成员

4月25日评论：

gr模型适配hstu使用ascendc融合算子实现，并未使用triton算子，请排查：

配套版本安装：配套版本 PyTorch torch-npu torchrec dynamic_emb fbgemm_gpu MindSpeed Megatron-LM

配套版本1 2.7.1 2.7.1 1.2.0+npu 25.09 1.2.0 core_v0.14.0 core_v0.14.0
patch是否成功打上

配套版本安装：配套版本	PyTorch	torch-npu	torchrec	dynamic_emb	fbgemm_gpu	MindSpeed	Megatron-LM
配套版本1	2.7.1	2.7.1	1.2.0+npu	25.09	1.2.0	core_v0.14.0	core_v0.14.0

taoqiuyu1998成员

5月11日评论：

/label add resolved

ascend-robot成员

5月11日添加了label：resolved

ascend-robot成员

5月19日评论：

您好，当前Issue标记为resolved且有一段时间未进一步更新，因此我们将其标记为'stale'（闲置）状态。若您认为这是误操作，可通过添加任意评论来去除'stale'标签。标记为stale的Issue在4天内无更新活动将自动关闭。

ascend-robot成员

5月19日添加了label：stale

ascend-robot成员

5月26日关闭了 issue

taoqiuyu1998成员

9月9日 issue状态由 TODO 改变为 DONE