你好,在使用triton-ascend的时候,无需安装triton。
注意到您同时安装了两个组件,可能会相互冲突导致不可预测的问题。
triton版本:3.5.0
triton-ascend版本:3.2.0
建议卸载后重新安装triton-ascend
pip uninstall triton triton-ascend
pip install triton-ascend


感谢解答!当前的问题已经通过您给出的方式解决了,但是遇到了一个新的问题:
[rank4]: Traceback (most recent call last):
[rank4]: File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 332, in
[rank4]: trainer = Trainer(args=args)
[rank4]: ^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 70, in init
[rank4]: self.model = self.get_model(model_provider)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 151, in get_model
[rank4]: model = self.get_foundation_model() if model_provider is None else model_provider()
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 138, in get_foundation_model
[rank4]: model = ModelHub.build(args.model, args.training)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/modelhub.py", line 124, in build
[rank4]: model: BaseModel = ModelHub._build_custom_model(model_args, training_args)
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/modelhub.py", line 41, in _build_custom_model
[rank4]: model = model_cls._from_config(model_args).float()
[rank4]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank4]: File "/root/transformers/src/transformers/modeling_utils.py", line 1444, in _from_config
[rank4]: dtype = kwargs.pop("dtype", config.dtype)
[rank4]: ^^^^^^^^^^^^
[rank4]: AttributeError: 'ModelArguments' object has no attribute 'dtype'
仍然是启动训练的时候出现的问题
我按照install_extensions.sh要求执行过:
cd transformers
git checkout fc91372


你好,请检查一下模型权重路径是否配置正确,大概率是路径错误导致的


好的感谢,问题已解决!


环境信息
使用场景及问题
在执行训练脚本:bash /data/user/chd/qwen3.5_27b_mindspeed_sft_workspace/finetune_qwen3_5_27B.sh操作时出错,显示:
Successfully imported: mindspeed_mm.fsdp.models.qwen3_5
Traceback (most recent call last):
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/utils.py", line 132, in get_available_device
return triton.runtime.driver.active.get_current_target().backend
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/driver.py", line 28, in active
self._active = self.default
^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/driver.py", line 22, in default
self._default = _create_driver()
^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/triton/runtime/driver.py", line 9, in _create_driver
raise RuntimeError(f"{len(active_drivers)} active drivers ({active_drivers}). There should only be one.")
RuntimeError: 0 active drivers ([]). There should only be one.
同时紧接着另一个报错:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):", line 1204, in _gcd_import", line 1176, in _find_and_load", line 1147, in _find_and_load_unlocked", line 690, in _load_unlocked", line 940, in exec_module", line 241, in _call_with_frames_removed
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 332, in
trainer = Trainer(args=args)
^^^^^^^^^^^^^^^^^^
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 60, in init
self.initialize()
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 126, in initialize
import_plugin(getattr(args.training, "plugin", []))
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 35, in import_plugin
_ = import_package(module_name, print_info=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/utils/register.py", line 12, in import_package
package = importlib.import_module(package_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "
File "
File "
File "
File "
File "
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/chunk_gated_delta_rule.py", line 9, in
from .triton.chunk_delta_h import chunk_gated_delta_rule_bwd_dhu, chunk_gated_delta_rule_fwd_h
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/chunk_delta_h.py", line 10, in
from .utils import prepare_chunk_indices, prepare_chunk_offsets, get_autotune_config,
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/utils.py", line 143, in
device = get_available_device() if get_available_device() != 'hip' else 'cuda'
^^^^^^^^^^^^^^^^^^^^^^
File "/root/MindSpeed-MM/mindspeed_mm/fsdp/models/qwen3_5/triton/utils.py", line 134, in get_available_device
_cpu_device_warning()
^^^^^^^^^^^^^^^^^^^
NameError: name '_cpu_device_warning' is not defined
(按照了install_extensions.sh安装了triton_ascend==3.2.0
transformers fc91372分支)
npu-smi info能正常显示8张npu的情况
没有找到解决该报错的方法
训练的配置文件在附件1402279ad2eb4424bc3c213393a4a0bd.txt