Tutel

Tutel MoE：一个优化版的混合专家实现，也是首个面向具有动态行为的现代训练与推理场景，提出 "No-penalty Parallism/Sparsity/Capacity/.. Switching" 的并行方案。支持基于 MoE 的 GLM-5.x / DeepSeek-3.x / Kimi-2.x / Kimi-3.x / Qwen3 / Gpt-OSS 在 A100/A800/H100/MI300/.. 上直接进行 NVFP4/MXFP4/BlockwiseFP8 推理。

[模型下载]
  pip3 install -U "huggingface_hub[cli]" --upgrade

  # Kimi K3 模型：
  hf download --local-dir moonshotai/Kimi-K3 moonshotai/Kimi-K3
  hf download --local-dir modal-labs/Kimi-K3-DFlash modal-labs/Kimi-K3-DFlash

  # GLM-5.3 模型：
  hf download --local-dir Inferact/GLM-5.3-NVFP4 Inferact/GLM-5.3-NVFP4

  # GLM-5.3-Flash 模型：
  hf download --local-dir zai-org/GLM-5.3-Flash zai-org/GLM-5.3-Flash

[ND_MI300_192G_v5：提供 Kimi-K3/GLM-5/5.1/5.2 服务（适用于 Azure MI300x8 PCIe）]
  docker run -e WORKER=1 -e LOCAL_SIZE=8 -p 8000:8000 -it --rm --ipc=host --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) \
      --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add=video \
      tutelgroup/deepseek-671b:mi300x8-chat-20260831 --serve=core \
        --try_path zai-org/GLM-5.3-Flash \
        --try_path Inferact/GLM-5.3-NVFP4 \
        --try_path moonshotai/Kimi-K3 \
        --max_seq_len 200000 \
        --thinking_effort high

[ND_A100_80G_v4：提供 GLM-5/5.1/5.2/5.3 服务（适用于 Azure A100x8/H100x8/B200x8 SXM）]
  docker run -e WORKER=1 -e LOCAL_SIZE=8 -p 8000:8000 -it --rm --ipc=host --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) \
      -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 --privileged \
      tutelgroup/deepseek-671b:a100x8-chat-20260707 --serve=core \
        --try_path Inferact/GLM-5.3-NVFP4 \
        --max_seq_len 200000

sudo apt-get install -y npm
sudo npm install -g @anthropic-ai/claude-code@2.1.197
cat > run_claude.sh <<EOF && chmod a+x run_claude.sh
mkdir -p config/
export ANTHROPIC_BASE_URL="http://0.0.0.0:8000"
export ANTHROPIC_API_KEY="sk-ant-api00-local-mock-key"
export CLAUDE_CONFIG_DIR="config"
export DISABLE_AUTOUPDATER=1
echo '{"customApiKeyResponses": {"approved": ["api00-local-mock-key"]}}' > config/.claude.json
claude
EOF

./run_claude.sh

winget install OpenJS.NodeJS.LTS
winget install --id Git.Git -e --source winget
npm install -g @anthropic-ai/claude-code@2.1.197
  (
    echo(@echo off
    echo(if not exist config mkdir config
    echo(set ANTHROPIC_BASE_URL=http://0.0.0.0:8000
    echo(set ANTHROPIC_API_KEY=sk-ant-api00-local-mock-key
    echo(set CLAUDE_CONFIG_DIR=config
    echo(set DISABLE_AUTOUPDATER=1
    echo(echo({"customApiKeyResponses": {"approved": ["api00-local-mock-key"]}} ^> config\.claude.json
    echo(claude
) > run_claude.bat

.\run_claude.bat

[模型下载]
  pip3 install -U "huggingface_hub[cli]" --upgrade
  hf download moonshotai/Kimi-K2.7-Code --local-dir moonshotai/Kimi-K2.7-Code
  hf download moonshotai/Kimi-K2.6 --local-dir moonshotai/Kimi-K2.6
  hf download nvidia/Kimi-K2.5-NVFP4 --local-dir nvidia/Kimi-K2.5-NVFP4
  hf download nvidia/Kimi-K2-Thinking-NVFP4 --local-dir nvidia/Kimi-K2-Thinking-NVFP4
  hf download nvidia/DeepSeek-V3.2-NVFP4 --local-dir nvidia/DeepSeek-V3.2-NVFP4

[DeepSeek V3.2 长上下文（适用于 Azure A100x8/H100x8/B200x8 SXM）]
  docker run -e LOCAL_SIZE=8 -e WORKER=1 -it --rm --ipc=host --net=host --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) -v /tmp:/tmp \
      -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 --privileged \
      tutelgroup/deepseek-671b:a100x8-chat-20260707 --serve=webui --listen_port 8000 \
        --try_path nvidia/GLM-5.2-NVFP4 \
        --try_path nvidia/GLM-5.1-NVFP4 \
        --try_path nvidia/GLM-5-NVFP4 \
        --try_path moonshotai/Kimi-K2.7-Code \
        --try_path moonshotai/Kimi-K2.6 \
        --try_path nvidia/Kimi-K2.5-NVFP4 \
        --try_path nvidia/Kimi-K2-Thinking-NVFP4 \
        --try_path nvidia/DeepSeek-V3.2-NVFP4 \
        --try_path nvidia/DeepSeek-R1-NVFP4 \
        --max_seq_len 16384

[DeepSeek V3.2 长上下文（适用于 Azure MI300x8 PCIe）]
  docker run -e LOCAL_SIZE=8 -e WORKER=1 -it --rm --ipc=host --net=host --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/kfd --device=/dev/dri --group-add=video \
      --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /:/host -w /host$(pwd) -v /tmp:/tmp \
      tutelgroup/deepseek-671b:mi300x8-chat-20260707 --serve=webui --listen_port 8000 \
        --try_path nvidia/GLM-5.2-NVFP4 \
        --try_path nvidia/GLM-5.1-NVFP4 \
        --try_path nvidia/GLM-5-NVFP4 \
        --try_path moonshotai/Kimi-K2.7-Code \
        --try_path moonshotai/Kimi-K2.6 \
        --try_path nvidia/Kimi-K2.5-NVFP4 \
        --try_path nvidia/Kimi-K2-Thinking-NVFP4 \
        --try_path nvidia/DeepSeek-V3.2-NVFP4 \
        --try_path nvidia/DeepSeek-R1-NVFP4 \
        --max_seq_len 1000000

[OpenAI/Ollama/直接请求]
  curl -N -X POST http://0.0.0.0:8000/chat -d '{"text": "Write a Python code of the Quicksort algorithm."}'
  python3 -m tutel.examples.oai_request_stream --url '0.0.0.0:8000' --prompt 'Write a Python code of the Quicksort algorithm.'

[用于 Web 浏览器的 Open-WebUI 地址]
  xdg-open http://0.0.0.0:8000

[模型下载]
  pip3 install -U "huggingface_hub[cli]" --upgrade
  hf download microsoft/VibeVoice-1.5B --local-dir microsoft/VibeVoice-1.5B
  hf download Qwen/Qwen2.5-1.5B --local-dir Qwen/Qwen2.5-1.5B

  hf download aoi-ot/VibeVoice-Large --local-dir microsoft/VibeVoice-Large
  hf download Qwen/Qwen2.5-7B --local-dir Qwen/Qwen2.5-7B

[Microsoft VibeVoice（仅限 ND_A100/H100/B200）]
docker run -e LOCAL_SIZE=1 -it --rm -p 8001:8000 --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) -v /tmp:/tmp \
      -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 --privileged \
      -e VOICES="https://homepages.inf.ed.ac.uk/htang2/notes/speech-samples/103-1240-0000.wav" \
      tutelgroup/deepseek-671b:a100x8-chat-20251222 --serve=core \
        --try_path ./microsoft/VibeVoice-1.5B \
        --try_path ./microsoft/VibeVoice-Large

[Microsoft VibeVoice（仅限 ND_MI300_192G_v5）]
docker run -e LOCAL_SIZE=1 -it --rm -p 8001:8000 --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/kfd --device=/dev/dri --group-add=video \
      --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /:/host -w /host$(pwd) -v /tmp:/tmp \
      -e VOICES="https://homepages.inf.ed.ac.uk/htang2/notes/speech-samples/103-1240-0000.wav" \
      tutelgroup/deepseek-671b:mi300x8-chat-20251222 --serve=core \
        --try_path ./microsoft/VibeVoice-1.5B \
        --try_path ./microsoft/VibeVoice-Large

[音频生成请求]
  curl -X POST http://0.0.0.0:8001/chat -d '{"text": "VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text."}' > sound_output.mp3

DeepSeek-MoE/Qwen3-MoE/KimiK2-MoE/GptOSS-MoE/.. 的推理 TPOS：

模型 & 机器类型	精度	SGL（无 MTP）	Tutel（无 MTP）
$deepseek-ai/DeepSeek-V3.2(671B,A100\times 8)$	nvfp4	-	102
$deepseek-ai/DeepSeek-V3.2(671B,MI300\times 8)$	nvfp4	-	151
$moonshotai/Kimi-K2-Instruct(1T,A100\times 8)$	nvfp4	-	104
$moonshotai/Kimi-K2-Instruct(1T,MI300\times 8)$	fp8b128	49	153
$NVFP4/Qwen3-235B-A22B-Instruct-2507-FP4(A100\times 8)$	nvfp4	-	114
$NVFP4/Qwen3-235B-A22B-Instruct-2507-FP4(MI300\times 8)$	nvfp4	-	122
$openai/gpt-oss-120b(120B,A100\times 1)$	mxfp4	127	212
$openai/gpt-oss-120b(120B,MI300\times 1)$	mxfp4	191	311
$microsoft/VibeVoice-1.5B(A100\times 1)$	bf16	-	rtf=0.07
$microsoft/VibeVoice-1.5B(MI300\times 1)$	bf16	-	rtf=0.06

What's New：

Image-20260831: 在 MI300 192GB PCIe-5 上支持 GLM-5.3/GLM-5.3-Flash。

Image-20260825: 在 MI300 192GB PCIe-5 上为 Kimi K3 新增 DFlash 支持。

Image-20260808: 在 MI300 192GB PCIe-5 上支持 Kimi K3 的 1M 上下文。

Image-20260707: 修复 A100 80GB SXM 上 1M 上下文相关的内存问题。

Image-20260618: 将 GLM-5.2 的 1M 上下文适配至 A100 80GB x 8 SXM。

Image-20260603: 优化 GLM-5/5.1 的 Claude Code Tooling 性能。

Image-20260511: 为 GLM-5/5.1 提供初始 Claude Code 支持。

Image-20260327: 新增对 Kimi-K2.5 的支持。

Image-20260306: 在 A100/H100/MI300/B200 上支持 DeepSeek V3.2 Long-context 模式。

Image-20251222: 针对大多数模型微调 A100 性能。

Image-20251111: 将 Tutel LLM 模块集成到 VibeVoice，以加速推理（单张 A100 的 rtf = 0.07）。

Image-20251006: 解决与 DeepSeek-V3.2-Exp 的兼容性问题

Image-20250827: 为 OpenAI GPT-OSS 20B/120B 新增分布式支持，并支持 MXFP4 推理

Image-20250801: 支持 Qwen3 MoE 系列，并集成 OpenWebUI

Image-20250712: 在 NVIDIA/AMD GPU 上支持使用 NVFP4 进行 Kimi K2 1TB MoE 推理

Image-20250601: 将 DeepSeek 671B 在 MI300x 上的解码性能提升至 140-150 TPS

更多镜像版本可在这里查看。

Tutel v0.4.2：为 NVIDIA 和 AMD GPU 新增 R1-FP4/Qwen3MoE-FP8 支持与 Fast Gating APIs：

  >> Example:

import torch
from tutel import ops

# Qwen3 Fast MoE Gating for 128 Experts, with Routed Weights normalized to 1.0
logits_fp32 = torch.softmax(torch.randn([32, 128]), -1, dtype=torch.float32).cuda()
topk_weights, topk_ids = ops.qwen3_moe_scaled_topk(logits_fp32)
print(topk_weights, topk_ids, topk_weights.sum(-1))

# DeepSeek V3/R1 Fast MoE Gating for 256 Experts, with Routed Weights normalized to 2.5
logits_bf16 = torch.randn([32, 256], dtype=torch.bfloat16).cuda()
correction_bias_bf16 = torch.randn([logits_bf16.size(-1)], dtype=torch.bfloat16).cuda()
topk_weights, topk_ids = ops.deepseek_moe_sigmoid_scaled_topk(logits_bf16, correction_bias_bf16, None, None)
print(topk_weights, topk_ids, topk_weights.sum(-1))

Tutel v0.4.1：支持在 NVIDIA GPU（A100 / A800）上运行 Deepseek R1 FP8 模型

Tutel v0.4.0：加速 AMD MI300x8 上 Deepseek R1 全精度对话体验：

  >> Example:

    # Step-1: Download Deepseek R1 671B Model
    huggingface-cli download deepseek-ai/DeepSeek-R1 --local-dir ./deepseek-ai/DeepSeek-R1

    # Step-2: Using 8 MI300 GPUs to Serve Deepseek R1 Chat on Local Port :8000
    docker run -it --rm --ipc=host --privileged -p 8000:8000 \
        -v /:/host -w /host$(pwd) tutelgroup/deepseek-671b:mi300x8-chat-20250224 \
        --model_path ./deepseek-ai/DeepSeek-R1

    # Step-3: Issue a Prompt Request with curl
    curl -X POST http://0.0.0.0:8000/chat -d '{"text": "Calculate the result of: 1 / (sqrt(5) - sqrt(3))"}'

Tutel v0.3.3：新增 all-to-all 基准测试：

  >> Example:

    python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.bandwidth_test --size_mb=256

Tutel v0.3.2：新增 tensorcore 选项以支持额外基准测试 / 扩展自定义专家示例 / 允许配置 NCCL 超时：

  >> Example of using tensorcore:

    python3 -m tutel.examples.helloworld --dtype=float32
    python3 -m tutel.examples.helloworld --dtype=float32 --use_tensorcore

    python3 -m tutel.examples.helloworld --dtype=float16
    python3 -m tutel.examples.helloworld --dtype=float16 --use_tensorcore

  >> Example of custom gates/experts:
    python3 -m tutel.examples.helloworld_custom_gate_expert --batch_size=16

  >> Example of NCCL timeout settings:
    TUTEL_GLOBAL_TIMEOUT_SEC=60 python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --use_tensorcore

Tutel v0.3.1：新增 NCCL all_to_all_v 与 all_gather_v，以支持任意长度消息传输：

  >> Example:
    # All_to_All_v:
    python3 -m torch.distributed.run --nproc_per_node=2 --master_port=7340 -m tutel.examples.nccl_all_to_all_v
    # All_Gather_v:
    python3 -m torch.distributed.run --nproc_per_node=2 --master_port=7340 -m tutel.examples.nccl_all_gather_v

  >> How to:
    net.batch_all_to_all_v([t_x_cuda, t_y_cuda, ..], common_send_counts)
    net.batch_all_gather_v([t_x_cuda, t_y_cuda, ..])

Tutel v0.3：新增 Megablocks 方案，用于在 num_local_expert >= 2 时提升单 GPU 上的解码器推理：

  >> Example (capacity_factor=0 required by dropless-MoE):
    # Using BatchMatmul:
    python3 -m tutel.examples.helloworld --megablocks_size=0 --batch_size=1 --num_tokens=32 --top=1 --eval --num_local_experts=128 --capacity_factor=0
    # Using Megablocks with block_size = 1:
    python3 -m tutel.examples.helloworld --megablocks_size=1 --batch_size=1 --num_tokens=32 --top=1 --eval --num_local_experts=128 --capacity_factor=0
    # Using Megablocks with block_size = 2:
    python3 -m tutel.examples.helloworld --megablocks_size=2 --batch_size=1 --num_tokens=32 --top=1 --eval --num_local_experts=128 --capacity_factor=0

  >> How to:
    self._moe_layer.forward(x, .., megablocks_size=1)         # Control the switch of megablocks_size (0 for disabled)

Tutel v0.2：支持以零成本动态切换大多数配置：

  >> Example:
    python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld_switch --batch_size=16

  >> How to:
    self._moe_layer.forward(x, .., a2a_ffn_overlap_degree=2)  # Control the switch of overlap granularity (1 for no overlapping)
    self._moe_layer.forward(x, .., adaptive_r=1)              # Control the switch of parallelism (0 for DP, 1 for DP + EP, W / E for MP + EP, else for DP + MP + EP)
    self._moe_layer.forward(x, .., capacity_factor=1)         # Control the switch of capacity_volume (positive for padding, negative for no-padding, 0 for dropless)
    self._moe_layer.forward(x, .., top_k=1)                   # Control the switch of top_k sparsity

Tutel v0.1：优化数据分发编码与解码的 Einsum 复杂度，新增 2DH 选项以应对大规模 All-to-All：

  >> Example (suggest enabling 2DH only at scale, note that the value of --nproc_per_node MUST equal to total physical GPU counts per node, e.g. 8 for A100x8):
    python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16 --use_2dh

快速开始

1. 准备 Pytorch（如适用）：

* Prepare Recommended Pytorch >= 2.0.0:
        #  Windows/Linux Pytorch for NVIDIA CUDA >= 11.7:
        python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
        #  Linux Pytorch for AMD ROCm >= 6.2.2:
        python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2.2
        #  Windows/Linux Pytorch for CPU:
        python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

2. Tutel 安装：

* Option-1: Install Tutel Online:

        $ python3 -m pip uninstall tutel -y
        $ python3 -m pip install -v -U --no-build-isolation git+https://github.com/microsoft/tutel@main

* Option-2: Build Tutel from Source:

        $ git clone https://github.com/microsoft/tutel --branch main
        $ python3 -m pip uninstall tutel -y
        $ python3 ./tutel/setup.py install --user

3. 单设备 / CPU 快速测试：

* Quick Test on Single-GPU:

        $ python3 -m tutel.examples.helloworld --batch_size=16               # Test Tutel-optimized MoE + manual distribution
        $ python3 -m tutel.examples.helloworld_ddp --batch_size=16           # Test Tutel-optimized MoE + Pytorch DDP distribution (requires: Pytorch >= 1.8.0)
        $ python3 -m tutel.examples.helloworld_ddp_tutel --batch_size=16     # Test Tutel-optimized MoE + Tutel DDP distribution (ZeRO on optimizors)
        $ python3 -m tutel.examples.helloworld_amp --batch_size=16           # Test Tutel-optimized MoE with AMP data type + manual distribution
        $ python3 -m tutel.examples.helloworld_custom_gate_expert --batch_size=16 # Test Tutel-optimized MoE + custom defined gate/expert layer
        $ python3 -m tutel.examples.helloworld_from_scratch                  # Test Custom MoE implementation from scratch
        $ python3 -m tutel.examples.moe_mnist                                # Test MoE layer in end-to-end MNIST dataset
        $ python3 -m tutel.examples.moe_cifar10                              # Test MoE layer in end-to-end CIFAR10 dataset

        (If building from source, the following method also works:)
        $ python3 ./tutel/examples/helloworld.py --batch_size=16
        ..

4. 单机 8 GPU 快速测试：

        $ python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16

5. 跨机器多 GPU 快速测试：

* Run Tutel MoE in Distributed Mode:

        (Option A - Torch launcher for `Multi-Node x Multi-GPU`:)
        $ ssh <node-ip-0> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr=<node-ip-0> -m tutel.examples.helloworld --batch_size=16
        $ ssh <node-ip-1> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr=<node-ip-0> -m tutel.examples.helloworld --batch_size=16

        (Option B - Tutel launcher for `Multi-Node x Multi-GPU`, requiring package `openmpi-bin`:)
        # << Single Node >>
        $ mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld_ddp_tutel --batch_size=16
        $ mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.moe_mnist
        $ mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.moe_cifar10
        ...

        # << MPI-based launch for GPU backend>>
        $ mpiexec -bind-to none -host <node-ip-0>,<node-ip-1>,.. -x MASTER_ADDR=<node-ip-0> -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16

        # << MPI-based Launch for CPU backend>>
        $ mpiexec -bind-to none -host localhost -x LOCAL_SIZE=1 -x OMP_NUM_THREADS=1024 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16 --device cpu

进阶：转换不同并行规模的检查点文件：

检查点转换文档已移至这里。

示例：如何在 Pytorch 中导入 Tutel 优化的 MoE：

# Input Example:
import torch
x = torch.ones([6, 1024], device='cuda:0')

# Create MoE:
from tutel import moe as tutel_moe
moe_layer = tutel_moe.moe_layer(
    gate_type={'type': 'top', 'k': 2},
    model_dim=x.shape[-1],
    experts={
        'num_experts_per_device': 2,
        'type': 'ffn', 'hidden_size_per_expert': 2048, 'activation_fn': lambda x: torch.nn.functional.relu(x)
    },
    scan_expert_func = lambda name, param: setattr(param, 'skip_allreduce', True),
)

# Cast to GPU
moe_layer = moe_layer.to('cuda:0')

# In distributed model, you need further skip doing allreduce on global parameters that have `skip_allreduce` mask, 
# e.g.
#    for p in moe_layer.parameters():
#        if hasattr(p, 'skip_allreduce'):
#            continue
#        dist.all_reduce(p.grad)


# Forward MoE:
y = moe_layer(x)

print(y)

参考文献

如需了解 Tutel 的更多技术细节，可参阅下方这篇论文：

@article {tutel,
author = {Changho Hwang and Wei Cui and Yifan Xiong and Ziyue Yang and Ze Liu and Han Hu and Zilong Wang and Rafael Salas and Jithin Jose and Prabhat Ram and Joe Chau and Peng Cheng and Fan Yang and Mao Yang and Yongqiang Xiong},
title = {Tutel: Adaptive Mixture-of-Experts at Scale},
year = {2022},
month = jun,
journal = {CoRR},
volume= {abs/2206.03382},
url = {https://arxiv.org/pdf/2206.03382.pdf},
}

MOELayer 的用法：

* Usage of MOELayer Args:

        gate_type        : dict-type gate description, e.g. {'type': 'top', 'k': 2, 'capacity_factor': -1.5, ..},
                              or a list of dict-type gate descriptions, e.g. [{'type': 'top', 'k', 2}, {'type': 'top', 'k', 2}],
                              the value of k in top-gating can be also negative, like -2, which indicates one GPU will hold 1/(-k) parameters of an expert
                              capacity_factor X can be positive (factor = X), zero (factor = max(needed_volumes)) or negative (factor = min(-X, max(needed_volumes))).
        model_dim        : the number of channels for MOE's input tensor
        experts          : a dict-type config for builtin expert network
        scan_expert_func : allow users to specify a lambda function to iterate each experts param, e.g. `scan_expert_func = lambda name, param: setattr(param, 'expert', True)`
        result_func      : allow users to specify a lambda function to format the MoE output and aux_loss, e.g. `result_func = lambda output: (output, output.l_aux)`
        group            : specify the explicit communication group of all_to_all
        seeds            : a tuple containing a tripple of int to specify manual seed of (shared params, local params, others params after MoE's)
        a2a_ffn_overlap_degree : the value to control a2a overlap depth, 1 by default for no overlap, 2 for overlap a2a with half gemm, ..
        parallel_type    : the parallel method to compute MoE, valid types: 'auto', 'data', 'model'
        pad_samples      : whether do auto padding on newly-coming input data to maximum data size in history

* Usage of dict-type Experts Config:

        num_experts_per_device : the number of local experts per device (by default, the value is 1 if not specified)
        hidden_size_per_expert : the hidden size between two linear layers for each expert (used for type == 'ffn' only)
        type             : available built-in experts implementation, e.g: ffn
        activation_fn    : the custom-defined activation function between two linear layers (used for type == 'ffn' only)
        has_fc1_bias     : If set to False, the expert bias parameters `batched_fc1_bias` is disabled. Default: True
        has_fc2_bias     : If set to False, the expert bias parameters `batched_fc2_bias` is disabled. Default: True

贡献

本项目欢迎贡献和建议。大多数贡献需要你同意一份贡献者许可协议（CLA），声明你有权并且确实愿意向我们授予使用你贡献的权利。详情请访问 https://cla.opensource.microsoft.com。

当你提交 pull request 时，CLA 机器人会自动判断你是否需要提供 CLA，并相应地标注该 pull request（例如状态检查、评论）。只需按照机器人提供的说明操作即可。在所有使用我们 CLA 的仓库中，你只需执行一次。

本项目已采用 Microsoft Open Source Code of Conduct。更多信息请参见 Code of Conduct FAQ 或就其他问题或意见联系 opencode@microsoft.com。

商标

本项目可能包含项目、产品或服务的商标或标识。对 Microsoft 商标或标识的授权使用必须遵守 Microsoft 商标与品牌指南。 在修改后的本项目版本中使用 Microsoft 商标或标识，不得引起混淆或暗示 Microsoft 的赞助。 使用任何第三方商标或标识均须遵守相关第三方的政策。