1.Describe the current behavior / 问题描述 (Mandatory / 必填)

在run_mode="predict", context.mode=0 (graph mode), context.device_target="CPU"的情况下
回自动设置MindSpore Context “infer_boost” 为 “on”, 从而导致部分算子如果运行在host侧会出现结果不对。

当前主要发现gather算子问题，出现在LlamaEmbedding中
例如如下脚本

"""llama3 predict example."""
import argparse
import os

import numpy as np
import mindspore as ms
from mindspore import Tensor, Model
import mindspore.common.dtype as mstype

from mindformers import MindFormerConfig
from mindformers.core.context import build_context
from mindformers.models.llama.llama_layer import LlamaEmbedding

def main(config_path, use_parallel, load_checkpoint, vocab_file):
    inputs = ["Hello"]
    batch_size = len(inputs)

    # init config with yaml
    config = MindFormerConfig(config_path)

    # init context
    build_context(config)


    tokens = np.array([[0, 1, 2]]).astype(np.int32)
    tokens = ms.Tensor(tokens)
    emb = LlamaEmbedding(128256, 4, mstype.float32)
    print(tokens)
    logits = emb(tokens)
    print(logits.shape)
    print(logits)
    print(emb.embedding_weight[0])
    print(emb.embedding_weight[1])
    print(emb.embedding_weight[2])
    return

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--config_path', type=str,
                        help='model config file path.', default="./cpu.yaml")
    parser.add_argument('--use_parallel', action='store_true',
                        help='if run model prediction in parallel mode.')
    parser.add_argument('--load_checkpoint', type=str,
                        help='load model checkpoint path or directory.')
    parser.add_argument('--vocab_file', type=str,
                        help='tokenizer model',default='tokenizer.model')

    args = parser.parse_args()
    main(
        args.config_path,
        args.use_parallel,
        args.load_checkpoint,
        args.vocab_file
    )

希望Embedding的输出结果与后续直接查询的权重一致，但是实际上返回的全部是第一个权重数据。

[[0 1 2]]
(1, 3, 4)
[[[ 0.00119806 -0.00945262 -0.00679049 -0.00180352]
  [ 0.00119806 -0.00945262 -0.00679049 -0.00180352]
  [ 0.00119806 -0.00945262 -0.00679049 -0.00180352]]]
[ 0.00119806 -0.00945262 -0.00679049 -0.00180352]
[0.01328395 0.01043057 0.00138017 0.00514018]
[ 0.00246564 -0.00739445  0.01570621  0.01958514]

附参考yaml文件

run_mode: 'predict'

use_parallel: False
# parallel context config
parallel:
  parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel
  gradients_mean: False
  enable_alltoall: False
  full_batch: True
  search_mode: "sharding_propagation"
  enable_parallel_optimizer: False
  strategy_ckpt_save_file: "./ckpt_strategy.ckpt"
  parallel_optimizer_config:
    gradient_accumulation_shard: False
    parallel_optimizer_threshold: 64

# mindspore context init config
context:
  mode: 0 #0--Graph Mode; 1--Pynative Mode
  device_target: "CPU"
  enable_graph_kernel: False
  max_call_depth: 10000
  max_device_memory: "23GB"
  save_graphs: False
  save_graphs_path: "./graph"
  device_id: 0

# model config
model:
  model_config:
    type: LlamaConfig
    batch_size: 1 # add for increase predict
    seq_length: 16
    hidden_size: 8
    num_layers: 1
    num_heads: 2
    n_kv_heads: 2
    vocab_size: 128256
    intermediate_size: 14336
    rms_norm_eps: 1.0e-5
    bos_token_id: 128000
    eos_token_id: 128001
    pad_token_id: 128002
    ignore_token_id: -100
    max_position_embedding: 131072
    compute_dtype: "float32"
    layernorm_compute_type: "float32"
    softmax_compute_type: "float32"
    rotary_dtype: "float32"
    param_init_type: "float32"
    use_past: True
    is_dynamic: True
    theta: 500000
    extend_method: "LLAMA3" # support "None", "PI", "NTK", "LLAMA3"
    scaling_factor:
     factor: 8.0
     low_freq_factor: 1.0
     high_freq_factor: 4.0
     original_max_position_embeddings: 8192
    use_flash_attention: False # FA can accelerate training or finetune
    offset: 0
    fine_grain_interleave: 1
    checkpoint_name_or_path: ""
    repetition_penalty: 1
    max_decode_length: 512
    block_size: 16
    num_blocks: 512
    top_k: 3
    top_p: 1
    do_sample: False
  arch:
    type: LlamaForCausalLM

2.Environment / 环境信息 (Mandatory / 必填)

• Hardware Environment / 硬件环境(Mandatory / 必填):
  Hardware (e.g.Ascend910B1/Mac CPU)

  样例：

• Software Environment / 软件环境 (Mandatory / 必填):
  迭代版本新增问题样例：（根据实际修改和增删）

3.Related testcase / 关联用例 (Mandatory / 必填)

见问题描述

4.Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

运行脚本

5.Describe the expected behavior / 预期结果 (Mandatory / 必填)

在pynative模式下运行结果正常，如下

[[0 1 2]]
(1, 3, 4)
[[[ 0.00113672 -0.00517879  0.00259764 -0.00793262]
  [-0.00979616  0.00536977  0.00073413  0.00601738]
  [ 0.00859051 -0.01486695  0.00160852  0.00306318]]]
[ 0.00113672 -0.00517879  0.00259764 -0.00793262]
[-0.00979616  0.00536977  0.00073413  0.00601738]
[ 0.00859051 -0.01486695  0.00160852  0.00306318]

6.Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

7.Special notes for this issue/备注 (Optional / 选填)