Closed
[Bug-Report|缺陷反馈]: torch_extension 接口导入缺少依赖时出现重复注册错误 #7140
yipyipAdd on  10 days agoClosed at  5 days ago
yipyipCollaborators
10 days ago Create

Describe the current behavior / 问题描述

同时安装常规 PyTorch 接口包与实验接口包后,在实验接口依赖的底层 Python 模块不可导入时,仅执行 import cann_ops_transformer,导入过程先出现缺少模块的错误,随后连续出现同一接口 schema 重复注册的错误,最终导致主包导入失败。

本次最小复现选择 ds41 的以下两个接口:

  • indexer_prologue_k_dsl:注册入口 torch.ops.cann_ops_transformer.ds41.indexer_prologue_k。
  • indexer_prologue_qw_dsl:注册入口 torch.ops.cann_ops_transformer.ds41.indexer_prologue_qw。

当前 DSL 工程目录表中的 ds41 组包括 mqsmla、qli、qsli、indexer_prologue_qw、indexer_prologue_k、engram_gate、attn_epilogue。本机最小复现仅选择 K、QW,不据此认定组内其他算子均有相同表现。

异常发生在 Python 导入阶段,未调用算子,未执行 NPU 计算。

Environment Information / 环境信息

  • 操作系统:Ubuntu 24.04.4 LTS,x86_64。
  • Python:3.12.15(conda 独立环境)。
  • PyTorch:2.12.0+cpu。
  • 本机安装环境:torch_npu 2.12.0、CANN 9.3.0、Ascend950DT_9582。
  • 接口包版本:1.0.0。
  • 定界方式:保留未修改版本的真实打包脚本、包加载器、OpBuilder 和 K/QW 接口代码,在隔离目录生成、解包接口 wheel,使用真实 PyTorch dispatcher 检查导入。为限定验证范围,最小复现中的 torch_npu 使用仅提供包路径的模块替身,底层模块缺失/可达由隔离探针控制;不调用设备计算。

Steps to reproduce the issue / 重现步骤

在完整环境中,对应接口打包和导入方式如下:

# 常规接口包
bash build.sh --torch_extension --incremental

# 实验接口包,选择 K、QW
bash build.sh --torch_extension --incremental \
  --experimental \
  --ops=indexer_prologue_k_dsl,indexer_prologue_qw_dsl \
  --vendor_name=experimental
  1. 安装上述两个接口 wheel。
  2. 使当前 Python 环境无法导入 ops.indexer_prologue_qw,例如尚未安装包含该模块的底层算子包。
  3. 在新的 Python 进程中执行:
import cann_ops_transformer
  1. 观察日志:先出现 No module named 'ops.indexer_prologue_qw',随后出现 cann_ops_transformer::ds41.indexer_prologue_qw 的 schema 重复注册错误,最终 import 失败。

本机隔离探针使用保留的原始接口包,在独立 Python 进程中重复两次,均复现该错误。仅提供可导入的模拟 QW 底层模块作对照时,两个接口均可访问,导入成功;对照没有调用实现函数。

Describe the expected behavior / 预期结果

接口包能够正常导入。如果缺少必需的底层依赖,应明确报告该依赖缺失。一次导入过程不应在缺少依赖后进一步出现相同 schema 重复注册错误,掩盖最初的异常。

触发条件与本机定界结果

以下链路已在保留的未修改版本上逐行跟踪,不涉及算子计算正确性,也不据此认定其他环境中的所有重复注册错误均由同一条件引起。

  1. 主包 cann_ops_transformer.ops 扫描 cann_ops_transformer.ops entry-point 分组,从中发现实验包提供的 K、QW 接口,因此仅 import 主包也会进入实验包。实验包的 ops 初始化使用同一套发现逻辑。

  2. 导入深层接口模块时,Python 先初始化父包。实验 wheel 生成的 ops/attention/__init__.py 依次执行:

    from .indexer_prologue_k_dsl import *
    from .indexer_prologue_qw_dsl import *
    
  3. 原始 QW 接口先执行 indexer_prologue_qw_op_builder._ensure_initialized(),注册 schema、Meta;随后在模块顶层执行:

    from ops.indexer_prologue_qw import indexer_prologue_qw as dsl_indexer_prologue_qw
    

    如果该底层模块不存在,QW 模块和 attention 父包初始化失败。已完成的 PyTorch 注册仍然保留,持有注册对象的 OpBuilder 模块已成功加载;Python 失败模块的缓存则不会作为成功导入保留。

  4. torch_extension/cann_ops_transformer/ops/__init__.py::_load_op 捕获首次 ImportError 并记录 warning,随后继续加载。实际跟踪中,第一次重复注册发生在循环继续加载 QW entry point、再次初始化 attention/QW 时。之后无条件尝试导入 graph-convert 子模块,又会触发失败父包的初始化。

  5. 原始公共注册器在 OpBuilder._ensure_initialized 中使用:

    hasattr(torch.ops.cann_ops_transformer, self.name)
    # self.name == "ds41.indexer_prologue_qw"
    

    该检查不会把点号拆成逐层属性访问。对已注册接口,实测:

    hasattr(torch.ops.cann_ops_transformer, "ds41.indexer_prologue_qw")  # False
    hasattr(torch.ops.cann_ops_transformer.ds41, "indexer_prologue_qw")  # True
    

    因此重导时仍进入 Library.define(schema),与第一次导入留下的 schema 冲突。

  6. entry-point 分支的 graph-convert 导入只捕获 ImportError;此时重复注册产生的 RuntimeError 会继续向外传播,最终导致主包 import 失败。

定界结论:本机复现的起点是 QW 底层模块不可导入;失败后再次初始化与已有 overload 注册判断失效共同导致后续重复注册。共用 torch.ops.cann_ops_transformer 本身不足以触发该错误。

以下为本机原始版本隔离复现的完整输出(保留具体接口名称、schema 和堆栈;路径为本机隔离验证目录):

Failed to load op 'indexer_prologue_k_dsl': No module named 'ops.indexer_prologue_qw'
Failed to load op 'indexer_prologue_qw_dsl': Tried to register an operator (cann_ops_transformer::ds41.indexer_prologue_qw(Tensor x, Tensor qr, Tensor wqb, Tensor ww, Tensor descale_qr, Tensor descale_wqb, Tensor rope_sin, Tensor rope_cos, *, float softmax_scale, Tensor? q=None, Tensor? descale_q=None, Tensor? w=None) -> (Tensor, Tensor, Tensor)) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34. Original registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34
Failed to load op 'indexer_prologue_k_dsl': Tried to register an operator (cann_ops_transformer::ds41.indexer_prologue_qw(Tensor x, Tensor qr, Tensor wqb, Tensor ww, Tensor descale_qr, Tensor descale_wqb, Tensor rope_sin, Tensor rope_cos, *, float softmax_scale, Tensor? q=None, Tensor? descale_q=None, Tensor? w=None) -> (Tensor, Tensor, Tensor)) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34. Original registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34
Failed to load op 'indexer_prologue_qw_dsl': Tried to register an operator (cann_ops_transformer::ds41.indexer_prologue_qw(Tensor x, Tensor qr, Tensor wqb, Tensor ww, Tensor descale_qr, Tensor descale_wqb, Tensor rope_sin, Tensor rope_cos, *, float softmax_scale, Tensor? q=None, Tensor? descale_q=None, Tensor? w=None) -> (Tensor, Tensor, Tensor)) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34. Original registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34
Failed to load op 'indexer_prologue_qw_dsl': Tried to register an operator (cann_ops_transformer::ds41.indexer_prologue_qw(Tensor x, Tensor qr, Tensor wqb, Tensor ww, Tensor descale_qr, Tensor descale_wqb, Tensor rope_sin, Tensor rope_cos, *, float softmax_scale, Tensor? q=None, Tensor? descale_q=None, Tensor? w=None) -> (Tensor, Tensor, Tensor)) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34. Original registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34
Failed to load op 'indexer_prologue_qw_dsl': Tried to register an operator (cann_ops_transformer::ds41.indexer_prologue_qw(Tensor x, Tensor qr, Tensor wqb, Tensor ww, Tensor descale_qr, Tensor descale_wqb, Tensor rope_sin, Tensor rope_cos, *, float softmax_scale, Tensor? q=None, Tensor? descale_q=None, Tensor? w=None) -> (Tensor, Tensor, Tensor)) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34. Original registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34
Traceback (most recent call last):
  File "/home/c00951894/temp/indexer-import-check-20261105/boundary_probe.py", line 23, in <module>
    import cann_ops_transformer
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer/__init__.py", line 18, in <module>
    from . import ops  # noqa: F401
    ^^^^^^^^^^^^^^^^^
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer/ops/__init__.py", line 110, in <module>
    _load_op(_name, _target)
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer/ops/__init__.py", line 76, in _load_op
    _gmod = importlib.import_module("%s.graph_convert_%s" % (module_path, name))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c00951894/.conda/envs/cyp/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/__init__.py", line 18, in <module>
    from . import ops  # noqa: F401
    ^^^^^^^^^^^^^^^^^
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/ops/__init__.py", line 110, in <module>
    _load_op(_name, _target)
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/ops/__init__.py", line 76, in _load_op
    _gmod = importlib.import_module("%s.graph_convert_%s" % (module_path, name))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c00951894/.conda/envs/cyp/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/ops/attention/__init__.py", line 2, in <module>
    from .indexer_prologue_qw_dsl import *
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/ops/attention/indexer_prologue_qw_dsl/__init__.py", line 16, in <module>
    from .indexer_prologue_qw import indexer_prologue_qw as _indexer_prologue_qw_impl  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/ops/attention/indexer_prologue_qw_dsl/indexer_prologue_qw.py", line 74, in <module>
    indexer_prologue_qw_op_builder._ensure_initialized()
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py", line 57, in _ensure_initialized
    self.register_schema(self.schema())
  File "/home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py", line 96, in register_schema
    get_as_library().define(schema)
  File "/home/c00951894/.conda/envs/cyp/lib/python3.12/site-packages/torch/library.py", line 163, in define
    result = self.m.define(schema, alias_analysis, tuple(tags))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Tried to register an operator (cann_ops_transformer::ds41.indexer_prologue_qw(Tensor x, Tensor qr, Tensor wqb, Tensor ww, Tensor descale_qr, Tensor descale_wqb, Tensor rope_sin, Tensor rope_cos, *, float softmax_scale, Tensor? q=None, Tensor? descale_q=None, Tensor? w=None) -> (Tensor, Tensor, Tensor)) with the same name and overload name multiple times. Each overload's schema should only be registered with a single call to def(). Duplicate registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34. Original registration: registered at /home/c00951894/temp/indexer-import-check-20261105/packages-1ba5kbu9/installed/cann_ops_transformer_experimental/op_builder/builder.py:34

仅补充可导入的模拟 QW 底层模块后的对照输出:

INTERFACE cann_ops_transformer.ds41.indexer_prologue_k
INTERFACE cann_ops_transformer.ds41.indexer_prologue_qw
IMPORT_OK_NO_KERNEL_LAUNCH

Special notes for this issue / 备注

  • 本条报告仅定界接口导入与注册行为,不包含算子精度、性能或 NPU 执行验收。
  • MQSMLA、QLI 的 DSL 接口也使用 ds41.<overload> 注册名,但其底层 ops 模块导入位于调用函数内部,正常 import 时不会触发本例的底层缺包条件;这不等于重复注册检查对它们一定正确。
  • 相关现象记录:#6994。本条补充本机最小复现、完整日志及已验证触发链路。
likedislike
YyipyipCollaborators
10 days ago Added label: bug-report
YyipyipCollaborators
10 days ago Modify description of issue
周云飞
周云飞Collaborators
10 days ago Comments:

问题说明

从您提供的日志和定界结果看,QW 缺少底层依赖后,导入失败但已注册的 schema 仍保留;后续重新初始化时,_ensure_initialized() 对 ds41.indexer_prologue_qw 的检查未识别已有注册,再次执行 Library.define(),最终重复注册异常掩盖了缺包错误。该结论限于本次 K/QW 导入场景,不涉及 NPU 计算,也不能推广到 ds41 全组。

处理建议

可以考虑修正公共 OpBuilder 对算子名及 overload 的检查,使重复初始化不重复注册;同时调整 _load_op,接口导入失败后不再继续加载其 graph-convert,并保留原始依赖异常。QW 的依赖检查也可前移至注册之前,减少失败后残留注册的情况。重点验证缺包时错误清晰、依赖可达时 K/QW 均可导入,避免仅靠捕获 RuntimeError 隐藏问题。

likedislike
周云飞周云飞Collaborators
9 days ago Assign Leeshoujem as responsible person
AlmaAlmaCollaborators
9 days ago Associated pull request
Alma
AlmaCollaborators
5 days ago Comments:

已修复 torch_extension 在缺底层依赖时,因带点号算子(如 ds41.indexer_prologue_qw)重复注册导致 import cann_ops_transformer 失败的问题。

原因:

OpBuilder 用 hasattr(torch.ops.cann_ops_transformer, "ds41.xxx") 判断是否已注册;点号名在 PyTorch 中是分层命名空间,该检查恒为 False,导入失败后 schema 残留再初始化会二次 Library.define。
entry-point 路径的 load_op 在接口 import 失败后仍继续加载 graph_convert*,再次触发失败模块初始化,放大为 Duplicate registration,掩盖原始缺依赖错误。

改动:

op_builder/builder.py:新增 is_op_registered(),按 . 逐层检查算子是否已注册。
ops/init.py:entry-point 加载失败时提前 return,不再拉 graph_convert
*。
缺依赖时应只报告依赖缺失;依赖齐全时 K/QW 等接口可正常导入,不再出现重复注册。

likedislike
AlmaAlmaCollaborators
5 days ago changed custom state from 进行中 to 已完成
AlmaAlmaCollaborators
5 days ago Close issue
CANN-robotCANN-robotCollaborators
5 days ago Added label: resolved