MindSpeed LLM Evaluation Support
Supported Benchmarks
MindSpeed LLM supports evaluating the accuracy of LLMs on public benchmark datasets. The supported benchmarks are listed below.
| Benchmark | Download Link | Validation Set | MindSpeed LLM | OpenCompass |
|---|---|---|---|---|
| MMLU | GitHub | test | 45.73% | 45.3% |
| CEval | HuggingFace | val | 33.87% | 32.5% |
| BoolQ | GitHub | dev | 75.44% | 74.9% |
| BBH | GitHub | test | 34.4% | 32.5% |
| AGIEval | GitHub | test | 20.6% | 20.6% |
| HumanEval | GitHub | test | 12.8% | 12.2% |
| CMMLU | Kaggle | test | -- | -- |
| GSM8k | GitHub | -- | -- | -- |
| HellaSwag | GitHub | -- | -- | -- |
| NeedleBench | HuggingFace | -- | -- | -- |
The following table lists the supported model evaluation statistics for MindSpeed LLM.
| Model | Task | MindSpeed LLM | Community Version | Model | Task | MindSpeed LLM | Community Version |
|---|---|---|---|---|---|---|---|
| Aquila-7B | BoolQ | 77.3% | -- | Aquila2-7B | BoolQ | 77.8% | -- |
| Aquila2-34B | BoolQ | 88.0% | -- | Baichuan-7B | BoolQ | 69.0% | 67.0% |
| Baichuan-13B | BoolQ | 74.7% | 73.6% | Baichuan2-7B | BoolQ | 70.0% | 63.2% |
| Baichuan2-13B | BoolQ | 78.0% | 67.0% | Bloom-7B | MMLU | 25.1% | -- |
| Bloom-176B | BoolQ | 64.5% | -- | ChatGLM3-6B | MMLU | 61.5% | -- |
| GLM4-9B | MMLU | 74.5% | 74.7% | CodeQwen1.5-7B | HumanEval | 54.8% | 51.8% |
| CodeLlama-34B | HumanEval | 48.8% | 48.8% | Gemma-2B | MMLU | 39.6% | -- |
| Gemma-7B | MMLU | 52.2% | -- | InternLM-7B | MMLU | 48.7% | 51.0% |
| Gemma2-9B | MMLU | 70.7% | 71.3% | Gemma2-27B | MMLU | 75.5% | 75.2% |
| Llama-7B | BoolQ | 74.6% | 75.4% | Llama-13B | BoolQ | 79.6% | 78.7% |
| Llama-33B | BoolQ | 83.2% | 83.1% | Llama-65B | BoolQ | 85.7% | 86.6% |
| Llama-2-7B | MMLU | 45.7% | -- | Llama-2-13B | BoolQ | 82.2% | 81.7% |
| Llama-2-34B | BoolQ | 82.0% | -- | Llama-2-70B | BoolQ | 86.4% | -- |
| Llama-3-8B | MMLU | 65.2% | -- | Llama-3-70B | BoolQ | 78.4% | -- |
| Llama-3.1-8B | MMLU | 65.3% | -- | Llama-3.1-70B | MMLU | 81.8% | -- |
| Llama-3.2-1B | MMLU | 31.8% | 32.2% | Llama-3.2-3B | MMLU | 56.3% | 58.0% |
| Mistral-7B | MMLU | 56.3% | -- | Mixtral-8x7B | MMLU | 70.6% | 70.6% |
| Mistral-8x22B | MMLU | 77% | 77.8% | MiniCPM-MoE-8x2B | BoolQ | 83.9% | -- |
| Qwen-7B | MMLU | 58.1% | 58.2% | Qwen-14B | MMLU | 65.3% | 66.3% |
| Qwen-72B | MMLU | 74.6% | 77.4% | Qwen1.5-0.5B | MMLU | 39.1% | -- |
| Qwen1.5-1.8B | MMLU | 46.2% | 46.8% | Qwen1.5-4B | MMLU | 59.0% | 56.1% |
| Qwen1.5-7B | MMLU | 60.3% | 61.0% | Qwen1.5-14B | MMLU | 67.3% | 67.6% |
| Qwen1.5-32B | MMLU | 72.5% | 73.4% | Qwen1.5-72B | MMLU | 76.4% | 77.5% |
| Qwen1.5-110B | MMLU | 80.4% | 80.4% | Yi-34B | MMLU | 76.3% | 75.8% |
| Qwen2-0.5B | MMLU | 44.6% | 45.4% | Qwen2-1.5B | MMLU | 54.7% | 56.5% |
| Qwen2-7B | MMLU | 70.3% | 70.3% | Qwen2-57B-A14B | MMLU | 75.6% | 76.5% |
| Qwen2-72B | MMLU | 83.6% | 84.2% | MiniCPM-2B | MMLU | 51.6% | 53.4% |
| DeepSeek-V2-Lite-16B | MMLU | 58.1% | 58.3% | Qwen2.5-0.5B | MMLU | 47.67% | 47.5% |
| Qwen2.5-1.5B | MMLU | 59.4% | 60.9% | Qwen2.5-3B | MMLU | 65.6% | 65.6% |
| Qwen2.5-7B | MMLU | 73.8% | 74.2% | Qwen2.5-14B | MMLU | 79.4% | 79.7% |
| Qwen2.5-32B | MMLU | 83.3% | 83.3% | Qwen2.5-72B | MMLU | 85.59% | 86.1% |
| InternLM2.5-1.8B | MMLU | 51.3% | 53.5% | InternLM2.5-7B | MMLU | 71.6% | 71.6% |
| InternLM2.5-20B | MMLU | 73.3% | 74.2% | InternLM3-8B | MMLU | 76.6% | 76.6% |
| Yi1.5-6B | MMLU | 63.2% | 63.5% | Yi1.5-9B | MMLU | 69.2% | 69.5% |
| Yi1.5-34B | MMLU | 76.9% | 77.1% | CodeQwen2.5-7B | HumanEval | 66.5% | 61.6% |
| Qwen2.5-Math-7B | MMLU-STEM | 67.8% | 67.8% | Qwen2.5-Math-72B | MMLU-STEM | 83.7% | 82.8% |
| MiniCPM3-4B | MMLU | 63.7% | 64.6% | Phi-3.5-mini-instruct | MMLU | 64.39% | 64.34% |
| Phi-3.5-MoE-instruct | MMLU | 78.5% | 78.9% | DeepSeek-Math-7B | MMLU-STEM | 56.5% | 56.5% |
| DeepSeek-V2.5 | MMLU | 79.3% | 80.6% | DeepSeek-V2-236B | MMLU | 78.1% | 78.5% |
| Llama-3.3-70B-Instruct | MMLU | 82.7% | -- | QwQ-32B | MMLU | 81.19% | -- |
Evaluation Guide
The MindSpeed LLM evaluation guide is available here: Evaluation Guide.