MindCluster
- MindCluster
- News
- Introduction
- Directory Structure
- Release Notes
- Compatibility Information
- Environment Deployment
- Quick Start
- Feature Introduction
- API Reference
- FAQs
- Security Statement
- Branch Maintenance Policy
- Version Maintenance Policy
- Disclaimer
- License
- Contribution Guide
- Suggestions and Feedback
News
- [2026.04.15]: 🚀 Support for fault post-processing policy configuration
- [2026.04.15]: 🚀 Support for soft partitioning of A2\A3 devices
- [2026.04.15]: 🚀 Support for switch affinity during inference
- [2026.04.15]: 🚀 verl supports asynchronous saving
- [2026.04.15]: 🚀 Modification to ClusterD heartbeat frequency
- [2026.04.15]: 🚀 Enhancement of RoCE network fault isolation and recovery
- [2026.04.15]: 🚀 Improved accuracy of manual chip isolation
- [2026.04.15]: 🚀 Support for affinity scheduling within Tiangong networking
- [2026.04.15]: 🚀 Optimization of the job information subscription interface
- [2026.04.15]: 🚀 Volcano log optimization
- [2026.04.15]: 🚀 Support for automatic de-isolation of isolated chips
- [2026.04.15]: 🚀 Support for statistics on causes of abnormal task scheduling
- [2026.04.15]: 🚀 Support for hard partitioning of A2/A3 devices
- [2026.04.15]: 🚀 NPU Exporter supports custom metrics reporting based on files
- [2026.04.15]: 🚀 NPU Exporter supports NPU utilization metrics
- [2026.04.15]: 🚀 Optimization of NPU hardware fault handling process during process-level rescheduling
- [2026.04.15]: 🚀 Enhancement of fault handling during process-level rescheduling
Introduction
MindCluster (AI cluster system software) is a deep learning component that supports NPU (Ascend AI Processor) training and inference, enabling the construction of full-process cluster operations and providing functions such as NPU cluster job scheduling, O&M monitoring, and fault recovery. By leveraging MindCluster, deep learning platform developers can minimize development efforts related to underlying resource scheduling and rapidly build their platforms.
Directory Structure
├─ build
└─ component
├─ ascend-common
│ ├─ api
│ │ ├─ ascend-operator
│ │ │ ├─ apis
│ │ │ │ └─ batch
│ │ │ │ └─ v1
│ │ │ └─ client
│ │ │ ├─ clientset
│ │ │ │ └─ versioned
│ │ │ │ ├─ scheme
│ │ │ │ └─ typed
│ │ │ │ └─ batch
│ │ │ │ └─ v1
│ │ │ ├─ informers
│ │ │ │ └─ externalversions
│ │ │ │ ├─ batch
│ │ │ │ │ └─ v1
│ │ │ │ └─ internalinterfaces
│ │ │ └─ listers
│ │ │ └─ batch
│ │ │ └─ v1
│ │ └─ slownet
│ ├─ common-utils
│ │ ├─ cache
│ │ ├─ ethtool
│ │ ├─ hwlog
│ │ ├─ limiter
│ │ ├─ rand
│ │ └─ utils
│ └─ devmanager
│ ├─ common
│ ├─ dcmi
│ └─ hccn
├─ ascend-device-plugin
│ ├─ build
│ ├─ doc
│ │ └─ figures
│ └─ pkg
│ ├─ common
│ ├─ device
│ │ └─ deviceswitch
│ ├─ kubeclient
│ ├─ next
│ │ └─ deviceswitch
│ │ └─ customname
│ └─ server
├─ ascend-docker-runtime
│ ├─ assets
│ ├─ build
│ │ ├─ libboundscheck
│ │ ├─ makeself-header
│ │ └─ scripts
│ ├─ cli
│ │ ├─ src
│ │ └─ test
│ │ ├─ dt
│ │ └─ dt_go
│ ├─ destroy
│ │ └─ src
│ ├─ hook
│ │ └─ process
│ ├─ install
│ │ └─ process
│ ├─ mindxcheckutils
│ ├─ opensource
│ ├─ output
│ ├─ platform
│ └─ runtime
│ ├─ dcmi
│ └─ process
├─ ascend-faultdiag-online
│ └─ pkg
│ ├─algo_src
│ │ ├─ netfault
│ │ └─ slownode
│ ├─ core
│ ├─ model
│ ├─ register
│ ├─ service
│ └─ utils
├─ ascend-faultdiag
│ ├─build
│ ├─platform
│ ├─src
│ │ ├─ascend_fd
│ │ │ ├─configuration
│ │ │ ├─controller
│ │ │ ├─lib
│ │ │ ├─model
│ │ │ ├─module
│ │ │ │ └─mindie_trace_parser
│ │ │ ├─pkg
│ │ │ │ ├─customize
│ │ │ │ │ ├─custom_config
│ │ │ │ │ └─custom_entity
│ │ │ │ ├─diag
│ │ │ │ │ ├─knowledge_graph
│ │ │ │ │ │ ├─kg_engine
│ │ │ │ │ │ │ ├─graph
│ │ │ │ │ │ │ └─model
│ │ │ │ │ ├─network_congestion
│ │ │ │ │ ├─node_anomaly
│ │ │ │ │ │ ├─npu_anomaly
│ │ │ │ │ │ └─resource_preemption
│ │ │ │ │ │ └─utils
│ │ │ │ │ └─root_cluster
│ │ │ │ ├─parse
│ │ │ │ │ ├─blacklist
│ │ │ │ │ ├─knowledge_graph
│ │ │ │ │ │ ├─parser
│ │ │ │ │ │ └─utils
│ │ │ │ │ ├─network_congestion
│ │ │ │ │ ├─node_anomaly
│ │ │ │ │ └─root_cluster
│ │ │ ├─sdk
│ │ │ ├─utils
│ │ │ │ ├─constant
│ │ │ │ ├─fast_parser
│ │ │ │ └─timehub
│ │ │ └─wrapper
│ ├─test
│ │ ├─custom_operation
│ │ ├─dt
│ │ └─st
│ ├─scripts
│ │ ├─exp_covert
│ │ │ └─exp_lib_dir
│ │ └─local_diag
│ └─toolkit_src
├─ ascend-for-volcano
│ ├─ build
│ ├─ common
│ ├─ k8s
│ └─ util
│ ├─ config
│ ├─ doc
│ └─ figures
│ ├─ internal
│ │ ├─ npu
│ │ │ ├─ ascend310
│ │ │ │ ├─ card310x4
│ │ │ │ └─ chip310x4
│ │ │ ├─ ascend310p
│ │ │ │ ├─ card310px2
│ │ │ │ ├─ chip310px2
│ │ │ │ └─ vnpu
│ │ │ ├─ ascend910
│ │ │ │ ├─ ascend910a3
│ │ │ │ │ ├─ module910a3x16
│ │ │ │ │ └─ superpod
│ │ │ │ ├─ ascend910b
│ │ │ │ │ ├─ module910bx16
│ │ │ │ │ └─ vnpu
│ │ │ │ └─ ascend910old
│ │ │ │ └─ module910x8
│ │ │ ├─ base
│ │ │ └─ vnpu
│ │ ├─ nslb
│ │ ├─ rescheduling
│ │ └─ test
│ ├─ output
│ ├─ plugin
│ ├─ test
│ └─ testdata
│ └─ tor
├─ ascend-operator
│ ├─ build
│ └─ pkg
│ ├─ api
│ │ └─ v1
│ ├─ controllers
│ │ ├─ scaling
│ │ └─ v1
│ ├─ ranktable
│ │ ├─ common
│ │ ├─ generator
│ │ ├─ utils
│ │ ├─ v1
│ │ └─ v1dot2
│ ├─ testtool
│ └─ utils
├─ clusterd
│ ├─ build
│ └─ pkg
├─ container-manager
│ ├─ build
│ └─ pkg
│ ├─ command
│ ├─ common
│ ├─ container
│ │ ├─ app
│ │ └─ domain
│ ├─ devmgr
│ ├─ fault
│ │ ├─ app
│ │ └─ domain
│ ├─ reset
│ │ ├─ app
│ │ └─ domain
│ └─ workflow
├─ noded
│ ├─ build
│ └─ pkg
├─ npu-exporter
│ ├─ build
│ ├─ cmd
│ ├─ collector
│ ├─ platforms
│ ├─ plugins
│ ├─ tuils
│ └─ versions
├─ taskd
│ ├─ build
│ ├─ plugins
│ ├─ taskd
│ ├─ tests
│ ├─ venv
│ └─ Scripts
Release Notes
For details about MindCluster version compatibility, see Version Compatibility Details
Compatibility Information
Frameworks supporting basic scheduling and resumable training features of MindCluster: Pytorch, MindSpore.
Environment Deployment
This part introduces the compilation and installation methods of MindCluster.
MindCluster Cluster Scheduling
Compilation
-
Pull the entire MindCluster source code, for example, place it in the
/homedirectory. -
Modify the
mind-cluster-versionfield in the component version configuration file/home/mind-cluster/build/service_config.inito the desired version. The default value is as follows:mind-cluster-version=6.0.0 -
Enter the
/home/mind-cluster/builddirectory and select a build script to execute.cd /home/mind-cluster/build dos2unix *.sh && chmod +x *.sh ./build_all.sh $GOPATH -
After execution, go to
/home/mind-cluster. The compiled files are generated in theoutputdirectory of each component.
Note: The Go version used here is 1.21.
Component Installation
Before installing and using cluster scheduling components, you need to understand their features in advance, and select the features to use and install the corresponding components accordingly.
MindCluster Ascend FaultDiag
MindCluster Ascend FaultDiag requires Python version 3.7 or later. Before installing MindCluster Ascend FaultDiag, check that the dependent Python version meets the requirement.
Compilation and Build
Environment Requirements
- Python >= 3.7.5
- scikit-learn >=1.3.0
- pandas >=1.3.5
- numpy >=1.21.6, < 2.0.0
- joblib >=1.2.0, < 1.5.0
- ply >=3.11
Build
Clone the repository first, then run the build script in the project root directory.
git clone https://gitcode.com/Ascend/mind-cluster.git
cd mind-cluster/component/ascend-faultdiag
bash build/build.sh
Component Installation
For details, see Installing MindCluster Ascend FaultDiag.
Quick Start
MindCluster Cluster Scheduling
A single Atlas 800T A2 training server (serving as both the management node and compute node) is used as an example to demonstrate how to install NodeD, Ascend Device Plugin, Ascend Docker Runtime, Volcano, ClusterD, and Ascend Operator, and how to use the full-NPU scheduling feature to quickly submit training tasks. For details, see the user guide.
MindCluster Ascend FaultDiag
The main functions of MindCluster Ascend FaultDiag (fault diagnosis tool) include: provides log cleaning and fault diagnosis capabilities, extracts key information from logs related to training and inference processes, and analyzes the root cause node and fault event based on the cleaned key information from all nodes in a cluster. For details, see the user guide.
Feature Introduction
This part describes the specific features of MindCluster.
MindCluster Cluster Scheduling
| Feature | Introduction | Released |
|---|---|---|
| Containerization | Containerization | ✅ |
| Resource Monitoring | Resource Monitoring | ✅ |
| Virtual Instance | Virtual Instance | ✅ |
| Basic Scheduling | Basic Scheduling | ✅ |
| Resumable Training | Resumable Training | ✅ |
| Appliance | Appliance | ✅ |
| Best Practices of MindIE Motor Inference Tasks | Best Practices of MindIE Motor Inference Tasks | ✅ |
| Best Practices of SGLang Inference Tasks | Best Practices of SGLang Inference Tasks | ✅ |
| Best Practices of vLLM Inference Tasks | Best Practices of vLLM Inference Tasks | ✅ |
MindCluster Ascend FaultDiag
| Feature | Introduction | Released |
|---|---|---|
| Log Cleaning and Dumping | Log Cleaning and Dumping | ✅ |
| Fault Diagnosis | Fault Diagnosis | ✅ |
| Single-Server Fault Diagnosis | Single-Server Fault Diagnosis | ✅ |
| SuperPoD Fault Diagnosis | SuperPoD Fault Diagnosis | ✅ |
| Service Flow Log Cleaning | Service Flow Log Cleaning | ✅ |
| Root Cause Node Cleaning and Diagnosis | Root Cause Node Cleaning and Diagnosis | ✅ |
| Fault Event Cleaning and Diagnosis | Fault Event Cleaning and Diagnosis | ✅ |
| Custom Configuration File | Custom Configuration File | ✅ |
API Reference
For the MindCluster cluster scheduling APIs, see API Reference.
For the MindCluster Ascend FaultDiag APIs, see API Reference.
FAQs
For MindCluster cluster scheduling-related FAQs, see FAQs.
For FAQs related to MindCluster Ascend FaultDiag, see FAQs.
Security Statement
MindCluster Cluster Scheduling
- Currently, components are deployed in a containerized manner. ServiceAccount is used as an authentication and authorization method, where the ServiceAccount token is displayed in plain text. You are advised to perform your own security hardening measures.
- Currently, components are deployed in a privileged container mode, which poses certain risks. You are advised to perform your own security hardening measures.
- For other security statements, see Security Statement
- For the communication matrix, see Communication Matrix
- For public URLs, see Public URLs
MindCluster Ascend FaultDiag
- For the security statement, see Security Statement
- For details about the Public URL, see: Public URLs
Branch Maintenance Policy
The maintenance phases for version branches are as follows:
| Status | Duration | Description |
|---|---|---|
| Planned | 1-3 months | Feature planning |
| Developing | 3 months | Develop new features and fix issues, with regular new version releases |
| Maintained | 3-12 months | Regular branches are maintained for 3 months, and long-term support branches for 12 months. Critical bugs are fixed, no new features are merged, and patch versions are released based on bug impact. |
| End of Life (EOL) | N/A | The branch no longer accepts any modifications. |
Version Maintenance Policy
| Version | Maintenance Policy | Current Status | Release Date | Future Status | EOL Date |
|---|---|---|---|---|---|
| master | Long-term support | Development | In-development branch, not released | - | |
| v7.3.0 | Long-term support | Maintenance | 2026-01-13 | 2026-12-30 | |
| v7.2.RC1 | Regular branch | Maintenance | 2025-10-25 | Expected to enter unmaintained status from 2026/1/25 |
2025-10-27 |
| v7.1.RC1 | Regular branch | EOL | 2025-07-24 | 2025-10-24 | |
| v7.0.RC1 | Regular branch | EOL | 2025-04-27 | 2025-07-27 | |
| v6.0.0 | Long-term support | Maintenance | 2024-12-31 | Expected to enter unmaintained status from 2025-12-31 |
|
| v6.0.RC3 | Regular branch | EOL | 2024-11-20 | 2025-02-20 | |
| v6.0.RC2 | Regular branch | EOL | 2024-11-20 | 2025-02-20 | |
| v6.0.RC1 | Regular branch | EOL | 2024-11-20 | 2025-02-20 | |
| v5.0.0 | Long-term support | EOL | 2023-11-20 | 2024-11-20 |
Disclaimer
- This repository contains multiple development branches, which may include unfinished, experimental, or untested features. Before official release, these branches should not be used in any production environment or projects that rely on business-critical operations. Please be sure to use our official release versions to ensure code stability and security. This project and its contributors shall not be held responsible for any issues, losses, or data corruption resulting from the use of development branches.
- For official versions, please refer to the release versions at https://gitcode.com/ascend/mind-cluster/releases.
License
MindCluster is licensed under the Apache 2.0 license.
The documentation under the docs directory is licensed under CC-BY 4.0.
Contribution Guide
- Before contributing, please sign the Open Project Contributor License Agreement (CLA).
- If you encounter a bug, please submit an issue.
- If you plan to contribute bug-fixes, please submit a PR.
- If you plan to contribute new features or functions, please create an issue to discuss with us first. In your issue, please describe the background/purpose of the requirement, the design approach, and the impact on existing APIs, etc. Submitting a PR without prior discussion may result in rejection, as the project's evolution direction may differ from your ideas.
- For a more detailed contribution process, please refer to the Contribution Guide.
Suggestions and Feedback
Welcome to contribute to the community. If you have any questions or suggestions, please submit an issue, and we will respond as soon as possible. Thank you for your support.