[master]: 增加共享专家资料说明和 MLA 测试脚本-MindSpeed-AtomGit

已合并

[master]: 增加共享专家资料说明和 MLA 测试脚本 #1326

AtomGit-Bot创建于 2024年11月14日从refs/pull/1326/head 合入到 master

[master]: 增加共享专家资料说明和 MLA 测试脚本 #1326

已合并

AtomGit-Bot创建于 2024年11月14日

从refs/pull/1326/head合入到master

共 5 个文件变更+210-35

MREADME.md+7-6

	@@ -177,12 +177,13 @@ MindSpeed特性由六大模块组成，分别为：megetron特性支持、并行
177	\| Gloo 存档落盘优化 \| [link](docs/features/hccl-replace-gloo.md) \|	177	\| Gloo 存档落盘优化 \| [link](docs/features/hccl-replace-gloo.md) \|
178		178
179	## 关键场景特性	179	## 关键场景特性
180	-\| 特性 \| 介绍 \|	180	+\| 特性 \| 介绍 \|
181	-\|------------------------------\|-----------------------------------------------------------\|	181	+\|---------------------------------\|------------------------------------------------------\|
182	-\| Megatron Mcore MoE \| [link](docs/features/megatron_moe/megatron-moe.md) \|	182	+\| Megatron Mcore MoE \| [link](docs/features/megatron_moe/megatron-moe.md) \|
183	-\| DeepSpeed MoE \| [link](docs/features/deepspeed_moe/deepspeed-moe.md) \|	183	+\| DeepSpeed MoE \| [link](docs/features/deepspeed_moe/deepspeed-moe.md) \|
184	-\| 【Prototype】Ascend alibi \| [link](docs/features/alibi.md) \|	184	+\| Ascend 共享专家 \| [link](docs/features/shared-experts.md) \|
185	-\| 【Prototype】Ascend EOD Reset训练场景 \| [link](docs/features/eod-reset.md) \|	185	+\| 【Prototype】Ascend alibi \| [link](docs/features/alibi.md) \|
		186	+\| 【Prototype】Ascend EOD Reset训练场景 \| [link](docs/features/eod-reset.md) \|
186		187
187	## 其它特性	188	## 其它特性
188	\| 特性 \| 介绍 \|	189	\| 特性 \| 介绍 \|

MSECURITYNOTE.md+30-29

Adocs/features/shared-experts.md+29-0

@@ -0,0 +1,29 @@
	1	+# 共享专家特性
	2	+
	3	+## 方案介绍：
	4	+
	5	+随着混合专家模型MoE的演进，产生了路由专家和共享专家的概念。针对路由专家，输入数据会经过路由模块选择概率较高的专家进行计算；而对于共享专家，输入数据无需经过路由模块计算，所有数据都会经过共享专家计算。路由专家和共享专家的计算结果相加后作为MoE模块最终的计算结果。
	6	+
	7	+通过将共享专家和路由专家结合，MOE模型能够在不同的输入情况下既关注到输入数据的共性也能关注到输入数据的差异性，从而提高模型的泛化能力。
	8	+
	9	+共享专家如下图c所示（参考论文：https://arxiv.org/pdf/2401.06066 ）：
	10	+![img](../../sources/images/shared-experts.png)
	11	+
	12	+## 使用场景
	13	+
	14	+MoE场景下使用
	15	+
	16	+## 使用方法
	17	+
	18	+共享专家相关命令和参数说明：
	19	+
	20	+\| 命令参数 \| 参数说明 \|
	21	+\|--------------------------\|------------------------\|
	22	+\| `--n-shared-experts [int]` \| 共享专家数量 \|
	23	+\| `--ffn-hidden-size [int]` \| 共享专家中间隐藏层大小（与路由专家相同参数） \|
	24	+
	25	+## 注意事项
	26	+
	27	+开启共享专家需要开启：`--use-mcore-models`
	28	+
	29	+

Asources/images/shared-experts.png+0-0

Atests_extend/system_tests/feature_tests/deepseek_mla.sh+144-0

@@ -0,0 +1,144 @@
	1	+#!/bin/bash
	2	+
	3	+export CUDA_DEVICE_MAX_CONNECTIONS=1
	4	+source "tests_extend/system_tests/env_npu.sh"
	5	+
	6	+NPUS_PER_NODE=8
	7	+MASTER_ADDR=localhost
	8	+MASTER_PORT=6001
	9	+NNODES=1
	10	+NODE_RANK=0
	11	+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
	12	+
	13	+CKPT_DIR=./ckpt_llama
	14	+DATA_PATH="/home/dataset/llama2/alpaca_text_document"
	15	+TOKENIZER_MODEL="/home/dataset/model/llama-2-7b-hf/tokenizer.model"
	16	+
	17	+TP=1 # MLA only support TP1
	18	+PP=2
	19	+CP=1
	20	+EP=2
	21	+
	22	+DISTRIBUTED_ARGS="
	23	+ --nproc_per_node $NPUS_PER_NODE \
	24	+ --nnodes $NNODES \
	25	+ --node_rank $NODE_RANK \
	26	+ --master_addr $MASTER_ADDR \
	27	+ --master_port $MASTER_PORT
	28	+"
	29	+
	30	+MOE_ARGS="
	31	+ --expert-model-parallel-size ${EP} \
	32	+ --moe-model-type megatron_moe \
	33	+ --moe-token-dispatcher-type alltoall \
	34	+ --moe-alltoall-overlap-comm \
	35	+ --moe-zero-memory level0 \
	36	+ --moe-tp-extend-ep \
	37	+ --moe-grouped-gemm \
	38	+ --moe-permutation-async-comm \
	39	+ --use-fused-moe-token-permute-and-unpermute \
	40	+ --n-shared-experts 1 \
	41	+ --num-experts 32 \
	42	+ --moe-router-topk 4 \
	43	+ --moe-aux-loss-coeff 0.02 \
	44	+"
	45	+
	46	+MLA_ARGS="
	47	+ --multi-head-latent-attention \
	48	+ --qk-rope-head-dim 64 \
	49	+ --qk-nope-head-dim 128 \
	50	+ --q-lora-rank 1536 \
	51	+ --kv-lora-rank 512 \
	52	+ --v-head-dim 128 \
	53	+ --qk-layernorm \
	54	+"
	55	+
	56	+ROPE_ARGS="
	57	+ --rope-scaling-beta-fast 32 \
	58	+ --rope-scaling-beta-slow 1 \
	59	+ --rope-scaling-factor 40 \
	60	+ --rope-scaling-mscale 0.707 \
	61	+ --rope-scaling-mscale-all-dim 0.707 \
	62	+ --rope-scaling-original-max-position-embeddings 4096 \
	63	+ --rope-scaling-type yarn
	64	+"
	65	+
	66	+GPT_ARGS="
	67	+ --tensor-model-parallel-size ${TP} \
	68	+ --pipeline-model-parallel-size ${PP} \
	69	+ --num-layers-per-virtual-pipeline-stage 1 \
	70	+ --use-mcore-models \
	71	+ --use-flash-attn \
	72	+ --use-fused-rotary-pos-emb \
	73	+ --use-fused-swiglu \
	74	+ --use-fused-rmsnorm \
	75	+ --use-ascend-mc2 \
	76	+ --sequence-parallel \
	77	+ --use-distributed-optimizer \
	78	+ --overlap-grad-reduce \
	79	+ --swap-attention \
	80	+ --num-layers 4 \
	81	+ --noop-layers 0,3 \
	82	+ --manual-gc \
	83	+ --manual-gc-interval 50 \
	84	+ --seq-length 8192 \
	85	+ --max-position-embeddings 8192 \
	86	+ --train-iters 10000 \
	87	+ --hidden-size 8192 \
	88	+ --num-attention-heads 128 \
	89	+ --ffn-hidden-size 4352 \
	90	+ --make-vocab-size-divisible-by 128 \
	91	+ --vocab-size 126464 \
	92	+ --micro-batch-size 1 \
	93	+ --global-batch-size 32 \
	94	+ --tokenizer-type Llama2Tokenizer \
	95	+ --tokenizer-model ${TOKENIZER_MODEL} \
	96	+ --disable-bias-linear \
	97	+ --lr-decay-style linear \
	98	+ --lr-warmup-iters 1500 \
	99	+ --short-seq-prob 0.0 \
	100	+ --attention-dropout 0.0 \
	101	+ --hidden-dropout 0.0 \
	102	+ --untie-embeddings-and-output-weights \
	103	+ --init-method-std 0.006 \
	104	+ --position-embedding-type rope \
	105	+ --normalization RMSNorm \
	106	+ --swiglu \
	107	+ --no-masked-softmax-fusion \
	108	+ --attention-softmax-in-fp32 \
	109	+ --no-gradient-accumulation-fusion \
	110	+ --bf16 \
	111	+ --group-query-attention \
	112	+ --num-query-groups 8 \
	113	+ --lr 2.0e-4 \
	114	+ --min-lr 2.0e-4 \
	115	+ --weight-decay 0.1 \
	116	+ --clip-grad 1.0 \
	117	+ --adam-beta1 0.9 \
	118	+ --adam-beta2 0.95 \
	119	+ --rotary-base 100000 \
	120	+ --norm-epsilon 1.0e-5 \
	121	+"
	122	+
	123	+DATA_ARGS="
	124	+ --data-path $DATA_PATH \
	125	+ --split 995,5,0
	126	+"
	127	+
	128	+OUTPUT_ARGS="
	129	+ --log-throughput \
	130	+ --log-interval 1 \
	131	+ --save-interval 10000 \
	132	+ --eval-interval 10000 \
	133	+ --eval-iters 10 \
	134	+"
	135	+
	136	+torchrun $DISTRIBUTED_ARGS pretrain_gpt.py \
	137	+ $GPT_ARGS \
	138	+ $MOE_ARGS \
	139	+ $MLA_ARGS \
	140	+ $ROPE_ARGS \
	141	+ $DATA_ARGS \
	142	+ $OUTPUT_ARGS \
	143	+
	144	+set +x

	@@ -34,35 +34,36 @@
34	## 公网地址声明	34	## 公网地址声明
35	- MindSpeed代码中包含公网地址声明如下表所示：	35	- MindSpeed代码中包含公网地址声明如下表所示：
36		36
37	-\| 类型 \| 开源代码地址 \| 文件名 \| 公网IP地址/公网URL地址/域名/邮箱地址 \| 用途说明 \|	37	+\| 类型 \| 开源代码地址 \| 文件名 \| 公网IP地址/公网URL地址/域名/邮箱地址 \| 用途说明 \|
38	-\| :------------: \|:------------------------------------------------------------------------------------------:\|:-----------------------------------------:\| :----------------------------------------------------------: \|:-----------------------------------------:\|	38	+\| :------------: \|:------------------------------------------------------------------------------------------:\|:----------------------------------------------------------:\| :----------------------------------------------------------: \|:-----------------------------------------:\|
39	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/gate.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| deepspeed moe源码地址 \|	39	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/gate.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| deepspeed moe源码地址 \|
40	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/gate.py \| https://arxiv.org/pdf/2006.16668.pdf \| 开源引入TopKGate类实现 \|	40	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/gate.py \| https://arxiv.org/pdf/2006.16668.pdf \| 开源引入TopKGate类实现 \|
41	-\| 开源引入 \| https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/moe.py \| mindspeed/moe/gate.py \| https://arxiv.org/pdf/2202.08906.pdf \| 开源引入apply_z_loss实现 \|	41	+\| 开源引入 \| https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/moe.py \| mindspeed/moe/gate.py \| https://arxiv.org/pdf/2202.08906.pdf \| 开源引入apply_z_loss实现 \|
42	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/moe_layer.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| deepspeed moe源码地址 \|	42	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/moe_layer.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| deepspeed moe源码地址 \|
43	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/moe_layer.py \| https://arxiv.org/pdf/2006.16668.pdf \| 开源引入MOELayer类实现 \|	43	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/moe_layer.py \| https://arxiv.org/pdf/2006.16668.pdf \| 开源引入MOELayer类实现 \|
44	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| mindspeed/moe/mixtral_parallel_mlpbm.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| deepspeed moe源码地址 \|	44	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| mindspeed/moe/mixtral_parallel_mlpbm.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| deepspeed moe源码地址 \|
45	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| mindspeed/moe/moe.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| deepspeed moe源码地址 \|	45	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| mindspeed/moe/moe.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/layer.py \| deepspeed moe源码地址 \|
46	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/utils.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| deepspeed moe源码地址 \|	46	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/utils.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| deepspeed moe源码地址 \|
47	-\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/transformer/moe/moe_utils.py \| mindspeed/moe/utils.py \| https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/transformer/moe/moe_utils.py \| megatron moe源码地址 \|	47	+\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/transformer/moe/moe_utils.py \| mindspeed/moe/utils.py \| https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/transformer/moe/moe_utils.py \| megatron moe源码地址 \|
48	-\| 开源引入 \| https://github.com/pytorch/pytorch/pull/40762 \| mindspeed/moe/utils.py \| https://github.com/pytorch/pytorch/pull/40762 \| alltoall实现源码 \|	48	+\| 开源引入 \| https://github.com/pytorch/pytorch/pull/40762 \| mindspeed/moe/utils.py \| https://github.com/pytorch/pytorch/pull/40762 \| alltoall实现源码 \|
49	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/utils.py \| https://arxiv.org/pdf/2006.16668.pdf \| einsum论文地址 \|	49	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/sharded_moe.py \| mindspeed/moe/utils.py \| https://arxiv.org/pdf/2006.16668.pdf \| einsum论文地址 \|
50	-\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/experts.py \| mindspeed/moe/experts.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/experts.py \| deepspeed moe源码地址 \|	50	+\| 开源引入 \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/experts.py \| mindspeed/moe/experts.py \| https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/moe/experts.py \| deepspeed moe源码地址 \|
51	-\| 开源引入 \| https://github.com/HazyResearch/flash-attention \| docs/features/flash-attention.md \| https://arxiv.org/pdf/2205.14135 \| flash-attention说明文档 \|	51	+\| 开源引入 \| https://github.com/HazyResearch/flash-attention \| docs/features/flash-attention.md \| https://arxiv.org/pdf/2205.14135 \| flash-attention说明文档 \|
52	-\| 开源引入 \| https://github.com/nvidia/megatron-lm \| docs/features/virtual-pipeline-parallel.md \| https://people.eecs.berkeley.edu/~matei/papers/2021/sc_megatron_lm.pdf \| virtual-pipeline-parallel说明文档 \|	52	+\| 开源引入 \| https://github.com/nvidia/megatron-lm \| docs/features/virtual-pipeline-parallel.md \| https://people.eecs.berkeley.edu/~matei/papers/2021/sc_megatron_lm.pdf \| virtual-pipeline-parallel说明文档 \|
53	-\| 开源引入 \| https://github.com/feifeibear/long-context-attention \| docs/features/hybrid-context-parallel.md \| https://arxiv.org/abs/2405.07719 \| hybrid-context-parallel说明文档 \|	53	+\| 开源引入 \| https://github.com/feifeibear/long-context-attention \| docs/features/hybrid-context-parallel.md \| https://arxiv.org/abs/2405.07719 \| hybrid-context-parallel说明文档 \|
54	-\| 开源引入 \| https://github.com/feifeibear/long-context-attention \| docs/features/ring-attention-context-parallel.md \| https://arxiv.org/pdf/2310.01889 \| ring-attention-context-parallel说明文档 \|	54	+\| 开源引入 \| https://github.com/feifeibear/long-context-attention \| docs/features/ring-attention-context-parallel.md \| https://arxiv.org/pdf/2310.01889 \| ring-attention-context-parallel说明文档 \|
55	-\| 开源引入 \| https://github.com/ofirpress/attention_with_linear_biases \| docs/features/alibi.md \| https://arxiv.org/pdf/2108.12409 \| alibi说明文档 \|	55	+\| 开源引入 \| https://github.com/ofirpress/attention_with_linear_biases \| docs/features/alibi.md \| https://arxiv.org/pdf/2108.12409 \| alibi说明文档 \|
56	-\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM \| docs/features/sequence-parallel.md \| https://arxiv.org/pdf/2205.05198 \| sequence-parallel说明文档 \|	56	+\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM \| docs/features/sequence-parallel.md \| https://arxiv.org/pdf/2205.05198 \| sequence-parallel说明文档 \|
57	-\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM \| docs/features/pipeline-parallel.md \| https://arxiv.org/pdf/1806.03377 \| pipeline-parallel说明文档 \|	57	+\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM \| docs/features/pipeline-parallel.md \| https://arxiv.org/pdf/1806.03377 \| pipeline-parallel说明文档 \|
58	-\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM/pull/598 \| docs/faq/data_helpers.md \| https://github.com/NVIDIA/Megatron-LM/pull/598 \| data_helpers说明文档 \|	58	+\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM/pull/598 \| docs/faq/data_helpers.md \| https://github.com/NVIDIA/Megatron-LM/pull/598 \| data_helpers说明文档 \|
59	-\| 开源引入 \| https://pytorch.org/docs/stable/distributed.html \| mindspeed/core/parallel_state.py \| https://pytorch.org/docs/stable/distributed.html \| torch.distributed相关接口注意事项 \|	59	+\| 开源引入 \| https://pytorch.org/docs/stable/distributed.html \| mindspeed/core/parallel_state.py \| https://pytorch.org/docs/stable/distributed.html \| torch.distributed相关接口注意事项 \|
60	-\| 开源引入 \| https://github.com/pytorch/pytorch/pull/40762 \| mindspeed/moe/utils.py \| https://github.com/pytorch/pytorch/pull/40762 \| _AllToAll自动反向参考 \|	60	+\| 开源引入 \| https://github.com/pytorch/pytorch/pull/40762 \| mindspeed/moe/utils.py \| https://github.com/pytorch/pytorch/pull/40762 \| _AllToAll自动反向参考 \|
61	-\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM \| mindspeed/optimizer/distrib_optimizer.py \| https://github.com/NVIDIA/Megatron-LM/blob/main/docs/source/distrib_optimizer.md \| distributed_optimizer_zero3_init文档字符串参数说明 \|	61	+\| 开源引入 \| https://github.com/NVIDIA/Megatron-LM \| mindspeed/optimizer/distrib_optimizer.py \| https://github.com/NVIDIA/Megatron-LM/blob/main/docs/source/distrib_optimizer.md \| distributed_optimizer_zero3_init文档字符串参数说明 \|
62	-\| 开源引入 \| https://github.com/InternLM/InternEvo \| mindspeed/docs/features/ring-attention-context-parallel.md \| https://arxiv.org/pdf/2406.18485 \| ring-attention-context-parallel说明文档 \|	62	+\| 开源引入 \| https://github.com/InternLM/InternEvo \| mindspeed/docs/features/ring-attention-context-parallel.md \| https://arxiv.org/pdf/2406.18485 \| ring-attention-context-parallel说明文档 \|
63	-\| 开源引入 \| https://github.com/sail-sg/zero-bubble-pipeline-parallelism \| mindspeed/docs/features/nanopipe-pipeline-parallel.md \| https://arxiv.org/abs/2401.10241 \| nanopipe-pipeline-parallel说明文档 \|	63	+\| 开源引入 \| https://github.com/sail-sg/zero-bubble-pipeline-parallelism \| mindspeed/docs/features/nanopipe-pipeline-parallel.md \| https://arxiv.org/abs/2401.10241 \| nanopipe-pipeline-parallel说明文档 \|
64	-\| 开源引入 \| https://github.com/iclr24-3434/AMPipe.git \| mindspeed/docs/features/ampipe.md \| https://openreview.net/pdf?id=yLgr02IsXY \| ampipe说明文档 \|	64	+\| 开源引入 \| https://github.com/iclr24-3434/AMPipe.git \| mindspeed/docs/features/ampipe.md \| https://openreview.net/pdf?id=yLgr02IsXY \| ampipe说明文档 \|
65	-\| 开源引入 \| https://gitee.com/ascend/pytorch \| mindspeed/docs/features/adaptive-recompute.md \| https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha001/apiref/envref/envref_07_0053.html \| 环境变量`PYTORCH_NPU_ALLOC_CONF`说明文档 \|	65	+\| 开源引入 \| https://gitee.com/ascend/pytorch \| mindspeed/docs/features/adaptive-recompute.md \| https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC2alpha001/apiref/envref/envref_07_0053.html \| 环境变量`PYTORCH_NPU_ALLOC_CONF`说明文档 \|
		66	+\| 开源引入 \| https://github.com/deepseek-ai/DeepSeek-MoE \| mindspeed/docs/features/shared-experts.md \| https://arxiv.org/pdf/2401.06066 \| 共享专家说明文档 \|
66		67
67		68
68	## 公开接口声明	69	## 公开接口声明