MindStudio Probe

English | 简体中文

✨ Latest News

🔹 [2026.03.28]: End-of-Life Notice for the ADump Module in the msProbe Repository
🔹 [2026.03.20]: Released Foundation Model Training Accuracy Debugging, Foundation Model Inference Accuracy Debugging, and Enabling Tools for Common Frameworks
🔹 [2025.12.31]: MindStudio Probe Completely Open-Sourced

ℹ️ Introduction

MindStudio Probe ( msProbe) is a full-scenario precision debugging toolchain built for Ascend AI processors, specifically designed for the precision debugging phase of model development. It supports mainstream frameworks such as PyTorch and MindSpore, significantly improving the efficiency of locating model precision issues.

⚙️ Features

Usage Scenario	Sub-mode/Sub-scenario	Feature Item	Feature Description	Reference Document
vLLM inference	Eager/Graph mode	Data collection	Complete the msProbe precision data collection.	Data Collection
		Data comparison	Perform precision comparison on the precision data dumped by msProbe to locate precision issues. Refer to hierarchical visual graph comparison or precision comparison modes.	Graph Comparison in Hierarchical Visualization Precision Comparison
	torchair	Data collection	Collect precision data via the set_ge_dump_config API.	Data Collection
		Precision comparison	Perform precision comparison on the precision data dumped by msProbe to locate precision issues.	Precision Comparison
	General scenario	Inference anomaly detection	Obtain vLLM inference output and detect anomalies.	Inference Anomaly Detection
SGLang inference	Eager mode	Data collection	Complete the msProbe precision data collection operation.	Data Collection
		Data comparison	Perform precision comparison on the precision data dumped by msProbe to locate precision issues.	Graph Comparison in Hierarchical Visualization Precision Comparison
ATB inference	-	Data collection	Collect precision data during ATB model execution by loading the ATB dump module before the ATB model runs.	Data Collection
		Precision comparison	Perform precision comparison on the ATB dump precision data to locate precision issues.	Precision Comparison
		Data conversion	Convert the ATB dump precision data into numpy (.npy) or PyTorch tensor (.pt) format files.	Data Conversion
Offline model inference	-	Data collection	Complete the msProbe precision data collection operation.	Data Collection
		Precision comparison	Provide a one-click offline model comparison feature; complete the comparison by simply inputting the model, without the need to collect data in advance, and quickly output results.	Precision Comparison
		Offline model data precision comparison	Provide offline model data comparison; perform precision comparison using the dump data of the offline model	Offline Model Data Precision Comparison
		Data conversion	Convert the offline model dump data into numpy (.npy) or PyTorch tensor (.pt) format files.	Data Conversion
PyTorch	Training scenario	Pre-training configuration check	Before training or precision comparison, compare configuration differences between two environments that may affect training precision.	Pre-training Configuration Check
		Data collection	Complete the msProbe precision data collection operation via config.json configuration.	Data Collection
		Precision pre-check	Scan all APIs in the training model running on Ascend NPUs and provide diagnosis and analysis of the precision status.	Precision Pre-check
		Graph comparison in hierarchical visualization	Parse the precision data dumped by msProbe, reconstruct the model graph structure, and compare precision data at various model levels.	Graph Comparison in Hierarchical Visualization
		Precision comparison	Perform precision comparison on the precision data dumped by msProbe to locate precision issues.	Precision Comparison
		Compilation precision comparison	Perform module-by-module precision comparison between eager mode and compile mode for models with torch.compile enabled, to identify forward, backward, or loss discrepancies introduced by compilation	Compilation Precision Comparison
		Training status monitoring	Collect and aggregate intermediate values of network layers, optimizers, and communication operators during model training to help diagnose anomalies in computation, communication, and optimizer during training.	Training Status Monitoring
		Checkpoint comparison	During or after training, compare two different checkpoints to evaluate model similarity.	Checkpoint Comparison
		Cross-network analysis of first overflowed node	Locate the first node where NaN or Inf appears in multi-rank scenarios using dump data.	Cross-Network Analysis of First Overflowed Node
		Trend visualization	Visualize the statistical data from msProbe data collection or training status monitoring across three dimensions: iteration steps, node rank, and tensor target.	Trend Visualization
	verl scenario	verl hyperparameter comparison and key hyperparameter verification	During or after verl training, compare the actual hyperparameter configurations collected from training logs on two different servers, or verify whether the configuration matches the key hyperparameter values, assisting users in efficiently comparing actual hyperparameter configurations and accelerating the identification of training precision issues caused by configuration discrepancies.	verl Hyperparameter Comparison and Key Hyperparameter Verification
		Asynchronous training-inference consistency comparison data collection	For verl ≥ v0.7.0, collect comparison data that ensures consistent input shapes during verl training-inference consistency comparison.	Collecting Data for Asynchronous verl Training-Inference Consistency Comparison
		verl training-inference consistency comparison data collection based on FSDP	For verl < v0.7.0 with FSDP training backend, collect comparison data that ensures consistent input shapes during verl training-inference consistency comparison.	Collecting Data for Verifying Data Consistency Between verl Training and Inference Based on FSDP
		verl Training-Inference Consistency Comparison Data Collection based on Megatron	For verl < v0.7.0 with Megatron training backend, collect comparison data that ensures consistent input shapes during verl training-inference consistency comparison	Collecting Data for Verifying Data Consistency Between verl Training and Inference Based on Megatron
		Precision comparison	Perform precision comparison on the precision data dumped by msProbe to locate precision issues.	Precision Comparison
		Training-inference consistency monitoring: token-Level probs_diff monitoring	Monitor training-inference consistency via probs_diff at the token level.	Training-Inference Consistency Monitoring: Token-Level probs_diff Monitoring
		verl Training-Inference Cross-Validation	Insert checkpoints at the end of two stages, replace the stage outputs, and determine whether the stage outputs are abnormal based on the training results after replacement	verl Training-Inference Cross-Validation
MindSpore training	-	Pre-training configuration check	Before training or precision comparison, compare configuration differences between two environments that may affect training precision.	Pre-training Configuration Check
		Data collection	Complete the msProbe precision data collection operation via config.json configuration.	Data Collection
		Precision pre-check	Scan all APIs in the training model running on Ascend NPUs and provide diagnosis and analysis of the precision status.	Precision Pre-check
		Graph comparison in hierarchical visualization	Parse the precision data dumped by msProbe, reconstruct the model graph structure, and compare precision data at various model levels	Graph comparison in hierarchical visualization
		Precision comparison	Perform precision comparison on the precision data dumped by msProbe to locate precision issues.	Precision Comparison
		Training status monitoring	Collect and aggregate intermediate values of network layers, optimizers, and communication operators during model training to help diagnose anomalies in computation, communication, and optimizer during training.	Training Status Monitoring
		Checkpoint comparison	During or after training, compare two different checkpoints to evaluate model similarity.	Checkpoint Comparison
		Trend visualization	Visualize the statistical data from msProbe data collection or training status monitoring across three dimensions: iteration steps, node rank, and tensor target.	Trend Visualization
MSAdapter scenario	-	Data collection	Complete the msProbe precision data collection operation via config.json configuration.	Data Collection
		Checkpoint comparison	During or after training, compare two different checkpoints to evaluate model similarity.	Checkpoint Comparison

🚀 Quick Start

For a quick one-stop experience of the complete workflow of data collection, NPU/GPU precision comparison, and graph comparison in hierarchical visualization mode in 10 minutes, see msProbe Quick Start.

📦 Installation Guide

msProbe supports three installation methods: PyPI, WHL, and compilation from source. For details, see msProbe Installation Guide.

📘 Usage Guide

msProbe's features cover various scenarios such as training and inference. Based on your actual usage scenario, select the corresponding feature item as mentioned above, and refer to the relevant documentation for configuration and usage.

💡 Typical Cases

🔹 Foundation Model Training Accuracy Debugging
🔹 Foundation Model Inference Accuracy Debugging
🔹 Enabling Tools for Common Frameworks

📚 Supplementary Materials

🔹 Precision Data Collection Baseline in PyTorch
🔹 Precision Pre-check Baseline in MindSpore
🔹 Precision Data Collection Baseline in MindSpore
🔹 Performance Baseline Report of the Training Status Monitoring Tool

❓ FAQs

For a summary of frequently asked questions and solutions, see FAQs.

🌌 Intelligent Search

To improve document search efficiency, we provide:
🔹 Exact Search (ReadTheDocs): Full-text keyword search, directly accessing information such as APIs, parameters, and error messages.
🔹 AI Q&A (DeepWiki): Natural language Q&A, quickly grasping the project architecture and module relationships.
🔹 AI Q&A (ZRead): Better Chinese Q&A experience, precisely locating feature usage and details.

🛠️ Contribution Guide

Welcome to contribute to this project. Please read Contribution Guide before you start.

⚖️ Related Notes

🔹 Release Notes
🔹 Developer Guide
🔹 Security Statement
🔹 Disclaimer
🔹 License Notice

🤝 Suggestions and Communication

We welcome everyone to contribute to the community. If you have any questions or suggestions, please submit issues, and we will respond as soon as possible. Thank you for your support.

Real-time Interaction (WeChat Group)	Official Updates (Official Account)	More Support (Assistant/Forum)
Scan the QR code to join the technical discussion group	Scan the QR code to follow the official account	Scan the QR codes to join the technical discussion group and follow the official account, to connect with the MindStudio user and developer community: Quick Q&A: Discuss technical issues in real time with community members Stay Updated: Receive version release and feature update notifications firsthand Share Experience: Exchange best practices and hands-on insights with fellow developers More Support Channels: 👉 Ascend Assistant: 👉 Ascend Forum:

🙏 Acknowledgments

This tool is jointly contributed by the following departments of Huawei:
🔹 Ascend Computing MindStudio Development Department
🔹 Parallel Distributed Computing Laboratory

Thank you for every PR from the community. Contributions to msProbe are welcome!