[Installation]: triton版本不兼容-MindSpeed-MM-AtomGit

已关闭

[Installation]: triton版本不兼容 #334

PolarisAB创建于 5月7日关闭于 7月1日

5月7日创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

环境信息

- 操作系统：openEuler 24.03
- 昇腾硬件信息  npu-smi 25.5.0   Version: 25.5.0   
- CANN软件版本  CANN	8.5.0
- 安装的对应软件版本 
Python	3.11
PyTorch	2.7.1
Torch NPU	2.7.1
mindspeed 0.12.1

您是如何安装操作的

# 依赖安装：
git clone https://gitcode.com/Ascend/MindSpeed-MM.git
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
git checkout core_v0.12.1
cp -r megatron ../MindSpeed-MM/
cd ..
cd MindSpeed-MM

git clone https://gitcode.com/Ascend/MindSpeed.git
cd MindSpeed
git checkout 93c45456c7044bacddebc5072316c01006c938f9
pip install -r requirements.txt
pip install -e .
cd ..
pip install -e .

git clone --branch 26.0.0 https://gitcode.com/Ascend/MindSpeed-MM.git
cd MindSpeed-MM
bash scripts/install.sh --msid eb10b92 && bash examples/qwen3_5/install_extensions.sh

完整报错信息：
(base) [root@B04-06-17U-AT800T-NODE-11 MindSpeed-MM]# bash examples/qwen3_5/finetune_qwen3_5_27B.sh
W0507 10:47:06.765000 1974 site-packages/torch/distributed/run.py:766]
W0507 10:47:06.765000 1974 site-packages/torch/distributed/run.py:766] *****************************************
W0507 10:47:06.765000 1974 site-packages/torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0507 10:47:06.765000 1974 site-packages/torch/distributed/run.py:766] *****************************************
/root/miniconda3/lib/python3.11/site-packages/torch_npu/dynamo/torchair/init.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Field 'use_triton_gdn' is not defined in ModelArguments. Adding as dynamic attribute.
Successfully imported: mindspeed_mm.fsdp.models.qwen3_5Successfully imported: mindspeed_mm.fsdp.models.qwen3_5

Successfully imported: mindspeed_mm.fsdp.models.qwen3_5
Successfully imported: mindspeed_mm.fsdp.models.qwen3_5
Traceback (most recent call last):
Traceback (most recent call last):
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 327, in
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 327, in
Traceback (most recent call last):
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 327, in
trainer = Trainer(args=args)trainer = Trainer(args=args)

trainer = Trainer(args=args)
^^ ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

^^
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 60, in init
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 60, in init
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 60, in init
self.initialize()
self.initialize()self.initialize()

File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 126, in initialize
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 126, in initialize
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 126, in initialize
import_plugin(getattr(args.training, "plugin", [])) import_plugin(getattr(args.training, "plugin", []))
import_plugin(getattr(args.training, "plugin", []))

File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 35, in import_plugin
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 35, in import_plugin
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 35, in import_plugin
_ = import_package(module_name, print_info=False)_ = import_package(module_name, print_info=False)

_ = import_package(module_name, print_info=False)
^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^
^
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 12, in import_package
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 12, in import_package
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 12, in import_package
package = importlib.import_module(package_name)
package = importlib.import_module(package_name)
package = importlib.import_module(package_name)
^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^
^
File "/root/miniconda3/lib/python3.11/importlib/init.py", line 126, in import_module
File "/root/miniconda3/lib/python3.11/importlib/init.py", line 126, in import_module
File "/root/miniconda3/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)return _bootstrap._gcd_import(name[level:], package, level)
return _bootstrap._gcd_import(name[level:], package, level)

                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "", line 1204, in _gcd_import
File "", line 1204, in _gcd_import
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1176, in _find_and_load
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 1147, in _find_and_load_unlocked
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 690, in _load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 940, in exec_module
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "", line 241, in _call_with_frames_removed
File "", line 241, in _call_with_frames_removed
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/chunk_gated_delta_rule.py", line 9, in
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/chunk_gated_delta_rule.py", line 9, in
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/chunk_gated_delta_rule.py", line 9, in
from .triton.chunk_delta_h import chunk_gated_delta_rule_bwd_dhu, chunk_gated_delta_rule_fwd_hfrom .triton.chunk_delta_h import chunk_gated_delta_rule_bwd_dhu, chunk_gated_delta_rule_fwd_hfrom .triton.chunk_delta_h import chunk_gated_delta_rule_bwd_dhu, chunk_gated_delta_rule_fwd_h

File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/chunk_delta_h.py", line 7, in
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/chunk_delta_h.py", line 7, in
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/chunk_delta_h.py", line 7, in
import tritonimport triton

import triton
File "/root/miniconda3/lib/python3.11/site-packages/triton/init.py", line 8, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/init.py", line 8, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/init.py", line 8, in
from .runtime import (from .runtime import (from .runtime import (

File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/init.py", line 1, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/init.py", line 1, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/init.py", line 1, in
from .autotuner import (Autotuner, Config, Heuristics, autotune, heuristics)from .autotuner import (Autotuner, Config, Heuristics, autotune, heuristics)

from .autotuner import (Autotuner, Config, Heuristics, autotune, heuristics)
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/autotuner.py", line 31, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/autotuner.py", line 31, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/autotuner.py", line 31, in
from .jit import KernelInterfacefrom .jit import KernelInterface

from .jit import KernelInterface

File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/jit.py", line 34, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/jit.py", line 34, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/jit.py", line 34, in
from ..runtime.driver import driverfrom ..runtime.driver import driver

from ..runtime.driver import driver

File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/driver.py", line 1, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/driver.py", line 1, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/driver.py", line 1, in
from ..backends import backends
from ..backends import backendsfrom ..backends import backends

File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 50, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 50, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 50, in
backends = _discover_backends()
backends = _discover_backends()backends = _discover_backends()

                                        ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 43, in _discover_backends
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 43, in _discover_backends
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 43, in _discover_backends
compiler = _load_module(name, os.path.join(root, name, 'compiler.py'))compiler = _load_module(name, os.path.join(root, name, 'compiler.py'))

compiler = _load_module(name, os.path.join(root, name, 'compiler.py'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^

File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 12, in _load_module
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 12, in _load_module
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/init.py", line 12, in _load_module
spec.loader.exec_module(module)spec.loader.exec_module(module)

spec.loader.exec_module(module)

File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/amd/compiler.py", line 1, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/amd/compiler.py", line 1, in
File "/root/miniconda3/lib/python3.11/site-packages/triton/backends/amd/compiler.py", line 1, in
from triton.backends.compiler import BaseBackend, GPUTarget, Language
from triton.backends.compiler import BaseBackend, GPUTarget, Languagefrom triton.backends.compiler import BaseBackend, GPUTarget, Language

ImportErrorImportErrorImportError: : cannot import name 'Language' from 'triton.backends.compiler' (/root/miniconda3/lib/python3.11/site-packages/triton/backends/compiler.py): cannot import name 'Language' from 'triton.backends.compiler' (/root/miniconda3/lib/python3.11/site-packages/triton/backends/compiler.py)
cannot import name 'Language' from 'triton.backends.compiler' (/root/miniconda3/lib/python3.11/site-packages/triton/backends/compiler.py)

上述安装过程全部根据官方指导文档操作，目前triton版本为3.5.1，请问是否有其他兼容版本

欢迎加入社区，感谢您对社区的贡献 🎉!

ascend-robot成员

5月7日添加了label：installation

PolarisAB

5月7日评论：

pip install triton==3.5.0降级到3.5.0后此报错已解决，但是出现新报错
Traceback (most recent call last):
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 327, in
trainer = Trainer(args=args)
^^^^^^^^^^^^^^^^^^
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 60, in init
self.initialize()
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 126, in initialize
import_plugin(getattr(args.training, "plugin", []))
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 35, in import_plugin
_ = import_package(module_name, print_info=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 12, in import_package
package = importlib.import_module(package_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/chunk_gated_delta_rule.py", line 9, in
from .triton.chunk_delta_h import chunk_gated_delta_rule_bwd_dhu, chunk_gated_delta_rule_fwd_h
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/chunk_delta_h.py", line 10, in
from .utils import prepare_chunk_indices, prepare_chunk_offsets, get_autotune_config,
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/utils.py", line 143, in
device = get_available_device() if get_available_device() != 'hip' else 'cuda'
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/utils.py", line 134, in get_available_device
_cpu_device_warning()
^^^^^^^^^^^^^^^^^^^
NameError: name '_cpu_device_warning' is not defined

PolarisAB

5月7日评论：

根据过往issue解决方案：
pip uninstall triton triton-ascend
pip install triton-ascend
已解决，现在新报错：
ImportError: cannot import name 'initialization' from 'transformers' (/root/miniconda3/lib/python3.11/site-packages/transformers/init.py)
_ = import_package(module_name, print_info=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 12, in import_package
package = importlib.import_module(package_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1204, in _gcd_import
File "", line 1176, in _find_and_load
File "", line 1147, in _find_and_load_unlocked
File "", line 690, in _load_unlocked
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/modeling_qwen3_5.py", line 29, in
from transformers import initialization as init

PolarisAB

5月7日评论：

更新transformers至最新版后解决

PolarisAB

5月7日评论：

卡在Converting format of dataset (num_proc=8): 100%|██████████| 157712/157712 [00:12<00:00, 12340.43 examples/s]
Rank 0, running tokenizer on train_dataset (num_proc=8): 0%| | 0/157712 [00:40<?, ? examples/s]这里
随后报错：
/root/miniconda3/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via init_process_group or barrier . Using the current device set by the user.
warnings.warn( # warn only once

PolarisAB

5月7日评论：

完整报错信息：Rank 0, running tokenizer on train_dataset: 0%| | 0/157712 [00:00<?, ? examples/s]
/root/miniconda3/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via init_process_group or barrier . Using the current device set by the user.
warnings.warn( # warn only once
[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 327, in
[rank0]: trainer = Trainer(args=args)
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 73, in init
[rank0]: self.train_dataloader = self.get_dataloader() if dataloader_provider is None else dataloader_provider(args)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 292, in get_dataloader
[rank0]: datasets = build_mm_dataset(data_config.dataset_param)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/data/init.py", line 25, in build_mm_dataset
[rank0]: return dataset_cls_or_func(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/data/datasets/huggingface/qwen2vl_dataset.py", line 141, in get_qwen2vl_dataset
[rank0]: train_dataset = train_dataset.map(
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 562, in wrapper
[rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3341, in map
[rank0]: for rank, done, content in Dataset._map_single(**unprocessed_kwargs):
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3697, in _map_single
[rank0]: for i, batch in iter_outputs(shard_iterable):
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3647, in iter_outputs
[rank0]: yield i, apply_function(example, i, offset=offset)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3570, in apply_function
[rank0]: processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/data/data_utils/func_utils/convert.py", line 547, in preprocess_dataset
[rank0]: input_ids, labels = self._encode_data_example(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/data/data_utils/func_utils/convert.py", line 491, in _encode_data_example
[rank0]: messages = self.template.mm_plugin.process_messages(prompt + response, images, videos, audios, self.processor)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/data/data_utils/func_utils/mm_plugin.py", line 617, in process_messages
[rank0]: self._validate_input(processor, images, videos, audios)
[rank0]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/data/data_utils/func_utils/mm_plugin.py", line 120, in _validate_input
[rank0]: raise ValueError("Processor was not found, please check and update your model file.")
[rank0]: ValueError: Processor was not found, please check and update your model file.
[Rank 3 | Local Rank 3] 2026-05-07 20:22:23,366 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
[Rank 2 | Local Rank 2] 2026-05-07 20:22:23,398 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
[Rank 5 | Local Rank 5] 2026-05-07 20:22:23,469 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
[Rank 1 | Local Rank 1] 2026-05-07 20:22:23,491 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
[Rank 4 | Local Rank 4] 2026-05-07 20:22:23,603 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
[Rank 6 | Local Rank 6] 2026-05-07 20:22:23,696 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"
[ERROR] 2026-05-07-20:22:23 (PID:9884, Device:0, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[Rank 7 | Local Rank 7] 2026-05-07 20:22:26,337 INFO [httpx:1025] => HTTP Request: HEAD https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/json/json.py "HTTP/1.1 200 OK"

PolarisAB

5月7日评论：

参照另一个issue,pip installed mistral-common-1.11.2后解决，供大家参考

PolarisAB

5月7日评论：

可以正确加载数据集了，但是报错[rank2]: AttributeError: 'int' object has no attribute 'shape' [rank0]: Traceback (most recent call last):
[rank0]:
[rank0]:
File "/opt/code_v26.0.e/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 328, in trainer. train()
[ranke]:
[ranke]:
File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 321, in train self.trainer.train()
[ranke]: [ranke]:
File "/opt/code_v26.0.8/MindSpeed-M/mindspeed_mm/fsdp/train/train_engine.py", Line 130, in train
Loss = self.train_step(train_dataloader_iter)

PolarisAB

5月8日评论：

贴一下完整报错信息：
[rank3]: Traceback (most recent call last):
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 328, in
[rank3]: trainer.train()
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 321, in train
[rank3]: self.trainer.train()
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/train_engine.py", line 130, in train
[rank3]: loss = self.train_step(train_dataloader_iter)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/train/train_engine.py", line 99, in train_step
[rank3]: output = self.model(**batch_data)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank3]: return inner()
[rank3]: ^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank3]: result = forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/utils/generic.py", line 900, in wrapper
[rank3]: output = func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/modeling_qwen3_5.py", line 2072, in forward
[rank3]: outputs = self.model(
[rank3]: ^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/utils/generic.py", line 900, in wrapper
[rank3]: output = func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/modeling_qwen3_5.py", line 1818, in forward
[rank3]: outputs = self.language_model(
[rank3]: ^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank3]: return inner()
[rank3]: ^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank3]: result = forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/utils/generic.py", line 976, in wrapper
[rank3]: output = func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/modeling_qwen3_5.py", line 1462, in forward
[rank3]: causal_mask = create_causal_mask(
[rank3]: ^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 171, in wrapped_func
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/masking_utils.py", line 939, in create_causal_mask
[rank3]: _preprocess_mask_arguments(config, inputs_embeds, attention_mask, past_key_values, position_ids, layer_idx)
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 171, in wrapped_func
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/lib/python3.11/site-packages/transformers/masking_utils.py", line 850, in _preprocess_mask_arguments
[rank3]: kv_length, kv_offset = past_key_values.get_mask_sizes(q_length, layer_idx)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/opt/code_v26.0.0/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/modeling_qwen3_5.py", line 198, in get_mask_sizes
[rank3]: query_length = cache_position.shape[0]
[rank3]: ^^^^^^^^^^^^^^^^^^^^
[rank3]: AttributeError: 'int' object has no attribute 'shape'

林

林明哲成员

5月9日评论：

关于开头的triton版本问题，我们在examples/qwen3_5/install_extensions.sh中，已经加上了指定3.2.0版本的依赖声明，如果按照正常安装步骤，应该可以正常安装。

这里有个小隐患：triton-ascend未来可能转向原生兼容的模式，但是当前和原生的triton存在冲突，如有冲突需要参考之前的issue卸载后重新安装。

后续我们也会完善文档说明，减少大家踩坑的可能

yaoyaoxu成员

7月1日将 yaoyaoxu 设为负责人

yaoyaoxu成员

7月1日评论：

针对您最后提到的问题，看起来是 transformers 版本适配问题。您 5 月 7 日更新最新版后解决，当时最新的transformers可能是5.3.0，建议更换为 Qwen3.5 指定的 5.2.0 版本再测试验证。
感谢反馈，我们先关闭该 issue，如有其他问题欢迎重新发起。

yaoyaoxu成员

7月1日 issue状态由 TODO 改变为 DONE

yaoyaoxu成员

7月1日关闭了 issue

ascend-robot成员

7月1日添加了label：resolved