您好，该问题会在使用2.2.0分支或2.3.0分支，使用非对应MindSpeed版本出现，使用对应MindSpeed版本即可修复。
如果使用的是2.3.0分支的MindSpeed-LLM，请使用2.3.0_core_r0.12.1分支的MindSpeed。

如果您目前没有其他问题，我们将根据流程关闭此 issue。
如后续有新的疑问或补充内容，欢迎随时重新打开本 issue，或另起一个新的 issue 进行提问。
感谢您的理解与支持！

我的MindSpeed，MindSpeed-LLM，Megatron版本能对应的上，但还是出现以上问题

mindspeed:

mindspeed-llm:

megatron:

(fly_mindspeed) root@msprof:/home/fly/workspace/MindSpeed-LLM# bash examples/mcore/qwen25/generate_qwen25_7b_ptd.sh
W0128 03:58:56.012000 3777429 site-packages/torch/distributed/run.py:766]
W0128 03:58:56.012000 3777429 site-packages/torch/distributed/run.py:766] *****************************************
W0128 03:58:56.012000 3777429 site-packages/torch/distributed/run.py:766] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0128 03:58:56.012000 3777429 site-packages/torch/distributed/run.py:766] *****************************************
/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:345: ImportWarning:
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.load, torch.set_default_device, torch.get_device_module, torch.sparse_compressed_tensor, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.Tensor.pin_memory, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************

  warnings.warn(msg, ImportWarning)
/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:276: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch_npu/dynamo/torchair/__init__.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:07 (PID:3777719, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:07 (PID:3777717, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:07 (PID:3777718, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:08 (PID:3777716, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:08 (PID:3777713, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:08 (PID:3777712, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:08 (PID:3777715, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
    module is deprecated and will be removed in 0.10.0. Please use
    'megatron.core.extensions.transformer_engine' instead.
  from megatron.core.transformer.custom_layers.transformer_engine import TENorm
Traceback (most recent call last):
  File "/home/fly/workspace/MindSpeed-LLM/inference.py", line 18, in <module>
    from mindspeed_llm import megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 86, in execute
    MindSpeedFeaturesManager.apply_features_patches(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
    feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/fly/workspace/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/transformer_block.py", line 44, in register_patches
    patch_manager.register_patch('megatron.core.transformer.transformer_block.TransformerBlock.forward',
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 181, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/mindspeed/patch_utils.py", line 75, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of forward exist !
[ERROR] 2026-01-28-03:59:08 (PID:3777714, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception
W0128 03:59:10.063000 3777429 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3777712 closing signal SIGTERM
W0128 03:59:10.063000 3777429 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3777713 closing signal SIGTERM
W0128 03:59:10.064000 3777429 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3777714 closing signal SIGTERM
W0128 03:59:10.064000 3777429 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3777715 closing signal SIGTERM
W0128 03:59:10.064000 3777429 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3777716 closing signal SIGTERM
E0128 03:59:10.480000 3777429 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 5 (pid: 3777717) of binary: /root/miniconda3/envs/fly_mindspeed/bin/python3.10
Traceback (most recent call last):
  File "/root/miniconda3/envs/fly_mindspeed/bin/torchrun", line 7, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in main
    run(args)
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch/distributed/run.py", line 883, in run
    elastic_launch(
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/fly_mindspeed/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
inference.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2026-01-28_03:59:10
  host      : msprof
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 3777718)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2026-01-28_03:59:10
  host      : msprof
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 3777719)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-01-28_03:59:10
  host      : msprof
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 3777717)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
[ERROR] 2026-01-28-03:59:10 (PID:3777429, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

确认是mindspeed版本问题，使用的conda环境中一直走的是master主线的mindspeed版本，由于修改了setup.py中的版本号未能卸载干净，重新使用新的conda环境切换到对应的2.3.0_core_r0.12.1版本mindspeed后成功跑通。
如果您目前没有其他问题，我们将根据流程关闭此 issue。
如后续有新的疑问或补充内容，欢迎随时重新打开本 issue，或另起一个新的 issue 进行提问。
感谢您的理解与支持！