使用HyperParallel进行LlamaFactory FSDP2 qwen3vl-8b sft报错RuntimeError: Environment variables RANK_ID has not been exported, please export variables 'RANK

已开启

使用HyperParallel进行LlamaFactory FSDP2 qwen3vl-8b sft报错RuntimeError: Environment variables RANK_ID has not been exported, please export variables 'RANK_ID' #116

jacka123创建于 5月8日

jacka123

5月8日创建

版本信息

参考文档：https://blog.llamafactory.net/posts/mindspore-hyperparallel-quickstart/
不使能hyper_parallel时可正常训练
llamafactory version: 0.9.5.dev0
Platform: Linux-4.19.90-vhulk2107.1.0.h699.eulerosv2r10.aarch64-aarch64-with-glibc2.34
Python version: 3.12.10
PyTorch version: 2.7.1+cpu (NPU)
Transformers version: 5.2.0
Datasets version: 4.0.0
Accelerate version: 1.11.0
PEFT version: 0.18.1
NPU type: Ascend910B1
CANN version: 9.0.0
TRL version: 0.24.0
Default data directory: detected
mindspore: 2.9.0
hyper_parallel: master commit bfe13cf

重现步骤

fsdp2配置：examples/accelerate/fsdp2_config.yaml

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_version: 2
machine_rank: 0
main_training_function: main
mixed_precision: bf16  # or fp16
num_machines: 1  # the number of nodes
num_processes: 8  # the number of GPUs in all nodes
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

模型配置：examples/ascend/qwen3vlmoe_full_sft_fsdp2.yaml

model_name_or_path: Qwen/Qwen3-VL-8B-Instruct
image_max_pixels: 262144
video_max_pixels: 16384
trust_remote_code: true
use_v1_kernels: true
flash_attn: fa2

### method
stage: sft
do_train: true
finetuning_type: full
disable_gradient_checkpointing: false

### dataset
dataset: llava_1k_en, llava_1k_zh
template: qwen3_vl
cutoff_len: 1024
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### HyperParallel
use_hyper_parallel: true

### output
output_dir: saves/Qwen3-VL-8B-Instruct/full/sft
logging_steps: 1
save_steps: 500
max_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: true
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-4
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null
seed: 1234

启动命令：

export HCCL_CONNECT_TIMEOUT=3600
accelerate launch  --config_file examples/accelerate/fsdp2_config.yaml     src/train.py examples/ascend/qwen3vlmoe_full_sft_fsdp2.yaml

报错信息

[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/disk1/hyperparallel/LlamaFactory-main/src/train.py", line 28, in <module>
[rank0]:     main()
[rank0]:   File "/mnt/disk1/hyperparallel/LlamaFactory-main/src/train.py", line 19, in main
[rank0]:     run_exp()
[rank0]:   File "/mnt/disk1/hyperparallel/LlamaFactory-main/src/llamafactory/train/tuner.py", line 139, in run_exp
[rank0]:     _training_function(config={"args": args, "callbacks": callbacks})
[rank0]:   File "/mnt/disk1/hyperparallel/LlamaFactory-main/src/llamafactory/train/tuner.py", line 86, in _training_function
[rank0]:     run_sft_hp(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]:   File "/mnt/disk1/hyperparallel/LlamaFactory-main/src/llamafactory/train/hyper_parallel/workflow.py", line 143, in run_sft
[rank0]:     train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/integration/llamafactory/trainer.py", line 447, in train
[rank0]:     return super().train(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/transformers/trainer.py", line 1412, in train
[rank0]:     return inner_training_loop(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/transformers/trainer.py", line 1542, in _inner_training_loop
[rank0]:     model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
[rank0]:                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/accelerate/accelerator.py", line 1555, in prepare
[rank0]:     result = self._prepare_fsdp2(*args)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/accelerate/accelerator.py", line 1687, in _prepare_fsdp2
[rank0]:     model = fsdp2_prepare_model(self, model)
[rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/integration/llamafactory/trainer.py", line 309, in _hp_fsdp2_prepare_model
[rank0]:     return fsdp2_prepare_model(accelerator, model, hp_args)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/integration/llamafactory/utils.py", line 484, in fsdp2_prepare_model
[rank0]:     fsdp2_kwargs = _build_fsdp2_kwargs(accelerator, model, hp_args, fsdp2_plugin)
[rank0]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/integration/llamafactory/utils.py", line 328, in _build_fsdp2_kwargs
[rank0]:     mesh = _build_device_mesh(accelerator, hp_args)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/integration/llamafactory/utils.py", line 66, in _build_device_mesh
[rank0]:     return init_device_mesh(device_type, (world_size,), mesh_dim_names=("dp",))
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/core/dtensor/device_mesh.py", line 1311, in init_device_mesh
[rank0]:     platform.init_process_group()
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/hyper_parallel/platform/mindspore/platform.py", line 780, in init_process_group
[rank0]:     dist.init_process_group(backend=backend, init_method=init_method, timeout=timeout, world_size=world_size,
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/mindspore/_checkparam.py", line 1397, in wrapper
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/mindspore/mint/distributed/distributed.py", line 584, in init_process_group
[rank0]:     init(backend)
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/mindspore/communication/management.py", line 197, in init
[rank0]:     _check_parallel_envs()
[rank0]:   File "/home/miniconda3/envs/ci_312/lib/python3.12/site-packages/mindspore/communication/management.py", line 82, in _check_parallel_envs
[rank0]:     raise RuntimeError("Environment variables RANK_ID has not been exported, please export variables 'RANK_ID'.")
[rank0]: RuntimeError: Environment variables RANK_ID has not been exported, please export variables 'RANK_ID'.
[ERROR] 2028-10-11-07:20:26 (PID:1620052, Device:0, RankID:-1) ERR99999 UNKNOWN applicaiton exception

jacka123

5月8日问题后端类型由 [] 改变为 [Ascend]

jacka123

5月8日关联分支由 [] 改变为 [master]

jacka123

5月8日关联组件由 [] 改变为 [B-SIG-Parallel]

cuiyushi成员

6月2日评论：

久等了。看了下你的问题，报错的根因是代码走到了MindSpore后端。
HyperParallel同时支持MindSpore和Torch双后端。在未设置环境变量的情况下，会优先检测MindSpore，一旦环境中MindSpore可用就会走MindSpore后端，从而触发上述报错。而LlamaFactory目前需要使用Torch后端。
解决方法是显式指定后端为Torch：

export HYPER_PARALLEL_PLATFORM=torch

另外有两点环境适配需要注意：

在Ascend 910B1上训练时，需要额外安装与对应CANN、Torch版本匹配的torch-npu。
你当前使用的是transformers v5.2.0，而我们目前主要在transformers v4.57.1上做的验证，建议切到v4.57.1进行训练。

后续我们也会逐步支持更多Torch和transformers版本。如果切换后仍有问题，欢迎贴出新的报错日志。