[Bug]: dryrun 泛化pynative_deepseekv3_671b_31_layers.yaml 训练报错ValueError: For primitive[GatherD], the x shape must be greater than or equal to 8192, but got 4096-mindformers-AtomGit

已关闭

[Bug]: dryrun 泛化pynative_deepseekv3_671b_31_layers.yaml 训练报错ValueError: For primitive[GatherD], the x shape must be greater than or equal to 8192, but got 4096 #2397

zhanxiaojie111创建于 6月30日关闭于 7月3日

zhanxiaojie111

6月30日创建

Checklist

1. 我已经搜索过相关问题，但没有得到预期的帮助.
2. 最新版本中该错误尚未修复.
3. 请注意，如果您提交的Bug描述缺少相应的环境信息和最小可复现的demo，我们将很难复现和解决该问题，从而降低收到反馈的可能性，甚至该问题将被关闭.

🐞 问题详细描述

[Bug]:  dryrun 泛化pynative_deepseekv3_671b_31_layers.yaml 训练报错
报错信息： ValueError: For primitive[GatherD], the x shape must be greater than or equal to 8192, but got 4096
复现命令：export MS_SIMULATION_LEVEL=1;export RANK_ID=1673; msrun --worker_num=2048 --local_worker_num=1 --master_port=1704 --cluster_time_out=300 --join=True python run_mindformer.py --config=deepseekv3_1673_003.yaml --mode 1

yaml：
checkpoint:
  enable_save: false
  save_path: output/checkpoint
  save_max: 1
  save_interleaved_steps: 100
  no_save_optim: false
  async_save: false
  prefix: custom
  remove_redundancy: false
  load_path: ''
  load_balanced: false
  no_load_optim: true
  load_worker_number: 1
training:
  steps: 1000
  local_batch_size: 2
  global_batch_size: 512
  max_norm: 1.0
  seed: 42
  deterministic: true
optimizer:
  type: AdamW
  betas:
  - 0.9
  - 0.95
  eps: 1.0e-08
  weight_decay: 0.01
  use_fused: true
lr_scheduler:
  type: ConstantWarmUpLR
  learning_rate: 1.0e-05
  warmup_ratio: 0
parallelism:
  data_parallel_shard: 16
  data_parallel_shard_strategy: optim_grads_params
  tensor_parallel: 4
  sequence_parallel: false
  context_parallel: 1
  context_parallel_method: colossal
  expert_parallel: 1
  moe_token_dispatcher_type: alltoall
  pipeline_parallel: 4
  pipeline_parallel_schedule: 1f1b
  pipeline_parallel_interleave_num: 1
  pipeline_parallel_layers_per_stage:
  - 0-9
  - 10-19
  - 20-29
  - 30-39
  pipeline_parallel_microbatch_size: 1
recompute:
  mode: full
  full_recompute_layer:
  - 0
  - 1
  - 2
  - 3
  - 4
  - 5
  - 6
  - 7
  - 8
  - 9
  - 10
  - 11
  - 12
  - 13
  - 14
  - 15
  - 16
  - 17
  - 18
  - 19
  - 20
  - 21
  - 22
  - 23
  - 24
  - 25
  - 26
  - 27
  - 28
  - 29
  - 30
  - 31
  - 32
  - 33
  - 34
  - 35
  - 36
  - 37
  - 38
  - 39
train_dataset:
  dataloader:
    type: BlendedMegatronDatasetDataLoader
    datasets_type: GPTDataset
    sizes:
    - 8000
    - 0
    - 0
    config:
      split: 1, 0, 0
      seq_length: 4096
      eod_mask_loss: false
      reset_position_ids: false
      create_attention_mask: false
      reset_attention_mask: false
      create_compressed_eod_mask: false
      eod_pad_length: 128
      eod: 1
      pad: -1
      data_path:
      - '1'
      - /home/workspace/large_model_dataset/deepseek3/671b/wiki-103_4096/wiki_4096_text_document
    column_names:
    - input_ids
    - labels
    - loss_mask
    - position_ids
    shuffle: false
    python_multiprocessing: false
    dataset_dir: /home/workspace/large_model_dataset/wiki_data/wiki_megatron_4096/wiki_4096_text_document
  drop_remainder: true
  num_parallel_workers: 8
  prefetch_size: 1
  numa_enable: false
model:
  use_attn_mask_compression: false
  model_type: deepseek_v3
  architectures: DeepseekV3ForCausalLM
  offset: 0
  vocab_size: 129280
  seq_length: 4096
  hidden_size: 7168
  intermediate_size: 18432
  num_hidden_layers: 40
  max_position_embeddings: 163840
  hidden_act: fusedswiglu
  num_attention_heads: 128
  rms_norm_eps: 1.0e-06
  add_bias_linear: false
  use_flash_attention: true
  multi_latent_attention: true
  kv_lora_rank: 512
  q_lora_rank: 1536
  qk_rope_head_dim: 64
  qk_nope_head_dim: 128
  v_head_dim: 128
  qk_layernorm: true
  attention_dropout: 0.0
  hidden_dropout: 0.0
  params_dtype: float32
  compute_dtype: bfloat16
  layernorm_compute_dtype: float32
  softmax_compute_dtype: float32
  rotary_dtype: float32
  initializer_range: 0.02
  num_nextn_predict_layers: 0
  mtp_loss_scaling_factor: 0.3
  position_embedding_type: yarn
  scaling_factor: 40
  beta_fast: 32
  beta_slow: 1
  mscale: 1
  mscale_all_dim: 1
  rope_theta: 10000
  router_dense_type: float32
  gated_linear_unit: true
  moe_intermediate_size: 2048
  routed_scaling_factor: 2.5
  first_k_dense_replace: 3
  n_routed_experts: 64
  num_experts_per_tok: 8
  n_shared_experts: 1
  moe_permute_fusion: false
  moe_shared_expert_intermediate_size: 2048
  moe_grouped_gemm: true
  moe_router_load_balancing_type: seq_aux_loss
  moe_aux_loss_coeff: 0.0006
  scoring_func: sigmoid
  norm_topk_prob: true
  moe_router_enable_expert_bias: true
  moe_router_bias_update_rate: 0.001
  use_pad_tokens: true
  topk_group: 4
  n_group: 8
  moe_token_dispatcher_type: alltoall
profiler:
  enable_profiling: false
  output_path: ./profile
  start_step: 6
  end_step: 6
  profiler_level: 1
  profiler_rank:
  - 0
  profile_memory: true
rank_size: 2048

详细的环境信息描述

910B

其他辅助信息

报错信息如下：

Traceback (most recent call last):
  File "/data/code/mindformers/run_mindformer.py", line 243, in <module>
    main(args_)
  File "/data/code/mindformers/run_mindformer.py", line 66, in main
    trainer.train()
  File "/data/code/mindformers/mindformers/pynative/trainer/trainer.py", line 671, in train
    self._inner_train_loop()
  File "/data/code/mindformers/mindformers/pynative/trainer/trainer.py", line 783, in _inner_train_loop
    loss, grad_norm = self.training_pp_step(inputs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/pynative/trainer/trainer.py", line 941, in training_pp_step
    loss = self.compute_pp_loss(inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/pynative/trainer/trainer.py", line 999, in compute_pp_loss
    outputs = self.schedule.run(**inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code//hyper-parallel/hyper_parallel/core/pipeline_parallel/scheduler.py", line 587, in run
    self.run_microbatches(split_args, split_kwargs, losses)
  File "/data/code//hyper-parallel/hyper_parallel/core/pipeline_parallel/scheduler.py", line 968, in run_microbatches
    self._exec_step(cur_step, arg_mbs, kwarg_mbs, losses)
  File "/data/code//hyper-parallel/hyper_parallel/core/pipeline_parallel/scheduler.py", line 839, in _exec_step
    out = stage.forward_one_chunk(micro_index, arg_mbs[micro_index], kwarg_mbs[micro_index])
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code//hyper-parallel/hyper_parallel/platform/mindspore/pipeline_parallel/stage.py", line 164, in forward_one_chunk
    out, grad_fn = forward_and_gradfn(
                   ^^^^^^^^^^^^^^^^^^^
  File "/data/code//hyper-parallel/hyper_parallel/platform/mindspore/pipeline_parallel/backward.py", line 678, in forward_and_gradfn
    raise e
  File "/data/code//hyper-parallel/hyper_parallel/platform/mindspore/pipeline_parallel/backward.py", line 675, in forward_and_gradfn
    res = fn(*inputs, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1420, in __call__
    return self._complex_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1432, in _complex_call
    output = self._run_construct(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1113, in _run_construct
    return _run_construct_inner()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1098, in _run_construct_inner
    output = self._run_grad_construct(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1136, in _run_grad_construct
    output = self.construct(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/models/deepseek3/modeling_deepseek_v3_pynative.py", line 65, in construct
    return self.model(
           ^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1416, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/pynative/base_models/gpt/gpt_model.py", line 378, in construct
    loss = self.compute_language_model_loss(labels, logits, loss_mask)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/pynative/base_models/gpt/gpt_model.py", line 482, in compute_language_model_loss
    return self.loss(logits, labels, loss_mask)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1416, in __call__
    return self.construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/pynative/loss/loss.py", line 215, in construct
    loss_reduce = self.nll_loss(log_softmax, label)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1418, in __call__
    return self._run_construct(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1113, in _run_construct
    return _run_construct_inner()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1098, in _run_construct_inner
    output = self._run_grad_construct(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/nn/cell.py", line 1136, in _run_grad_construct
    output = self.construct(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/code/mindformers/mindformers/pynative/loss/loss.py", line 110, in construct
    return _NLLLoss.apply(log_softmax, label)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/envs/ci_312/lib/python3.12/site-packages/mindspore/common/_grad_function.py", line 66, in apply
    return super().apply(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: For primitive[GatherD], the x shape must be greater than or equal to 8192, but got 4096.