[Bug]: RuntimeError: Operator element_size does not contain parallel layout infer func. DTensor dispatch requires explicit layout inference registration. Please register a distributed operator for 'el-MindStudio-Probe-AtomGit

已关闭

[Bug]: RuntimeError: Operator element_size does not contain parallel layout infer func. DTensor dispatch requires explicit layout inference registration. Please register a distributed operator for 'el #579

zhaojichen创建于 9月4日关闭于 17 天前

zhaojichen

9月4日创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

API 令牌或密钥
密码或身份验证凭证
私有网址或接口地址
个人或机密数据
...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

例如：
- 操作系统 欧拉
- 昇腾硬件信息 910B
- CANN软件版本 cann 9.0
- 安装的对应软件版本 torch2.11, torch_npu 2.11.0

🐛 问题描述

pip install 安装dump工具，配置dump为md5执行训练
报错如下 Traceback (most recent call last):
File "/disk1/code/hyper_model/model_code/scripts/train_dit.py", line 43, in
main()
File "/disk1/code/hyper_model/model_code/scripts/train_dit.py", line 38, in main
trainer.train()
File "/disk1/code/hyper_model/model_code/hyper_parallel/auto_models/trainer/dit_trainer.py", line 328, in train
self.train_step(data_iterator)
^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/common/utils.py", line 159, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/pytorch/dump/debugger/precision_debugger.py", line 94, in stop
cls._run_with_reload(lambda instance: instance.service.stop())
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/pytorch/dump/debugger/precision_debugger.py", line 122, in _run_with_reload
action(instance)
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/pytorch/dump/debugger/precision_debugger.py", line 94, in
cls._run_with_reload(lambda instance: instance.service.stop())
^^^^^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/service.py", line 183, in stop
self._flush_dump_result()
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/service.py", line 397, in _flush_dump_result
self.data_collector.write_json()
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/data_dump/data_collector.py", line 117, in write_json
self.data_writer.write_json()
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/data_dump/json_writer.py", line 397, in write_json
crc32_result = self.flush_crc32_stack()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/data_dump/json_writer.py", line 176, in flush_crc32_stack
results = [f.result() for f in self.crc32_stack_list]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/data_dump/json_writer.py", line 176, in
results = [f.result() for f in self.crc32_stack_list]
^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/data_dump/data_processor/pytorch_processor.py", line 305, in compute_crc32_from_tensor
mv = PytorchDataProcessor.tensor_bytes_view_cpu(t)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk0/miniconda3/envs/wan_hyper/lib/python3.11/site-packages/msprobe/core/dump/data_dump/data_processor/pytorch_processor.py", line 258, in tensor_bytes_view_cpu
nbytes = t.numel() * t.element_size()
^^^^^^^^^^^^^^^^
File "/disk1/code/hyper_model/model_code/hyper_parallel/platform/torch/dtensor.py", line 85, in torch_function
out = _OP_DISPATCHER.dispatch(func, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk1/code/hyper_model/model_code/hyper_parallel/core/shard/_op_dispatch.py", line 1062, in dispatch
result = self._dispatch_layout_infer(op_name, op_call, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk1/code/hyper_model/model_code/hyper_parallel/core/shard/_op_dispatch.py", line 954, in _dispatch_layout_infer
return self._handle_unregistered_op(op_name, op_call, args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/disk1/code/hyper_model/model_code/hyper_parallel/core/shard/_op_dispatch.py", line 912, in _handle_unregistered_op
raise RuntimeError(
RuntimeError: Operator element_size does not contain parallel layout infer func. DTensor dispatch requires explicit layout inference registration. Please register a distributed operator for 'element_size' or use local tensors.
Exception ignored in atexit callback: <bound method DataCollector.write_json_at_exit of <msprobe.core.dump.data_dump.data_collector.DataCollector object at 0xfffe7a0d5810>>