Ascend Device Plugin

Chip Resources

mindx-dl-deviceinfo-<nodename>

The NPU information reported by Ascend Device Plugin is shown in Table 1.

Table 1 DeviceInfoCfg

Name	Meaning	Description
huawei.com/Ascend910	Marks the chip name information available on the current node. When multiple chips exist, they are concatenated with commas.	This field is currently being phased out and will no longer be presented in subsequent versions. By default, the available chips on a node are maintained by Volcano, and this field does not take effect. To make it take effect, you can modify the Volcano configuration parameter "self-maintain-available-card" to false. Atlas 350 PCIe card, Atlas 850 series hardware products, and Atlas 950 SuperPoD use huawei.com/npu as the parameter name.
huawei.com/Ascend910-NetworkUnhealthy	Marks the chip name information for chips with unhealthy networks on the current node. When multiple chips exist, they are concatenated with commas.	Atlas 350 PCIe card, Atlas 850 series hardware products, and Atlas 950 SuperPoD use huawei.com/npu-NetworkUnhealthy as the parameter name.
huawei.com/Ascend910-Unhealthy	Marks the chip name information for unhealthy chips on the current node. When multiple chips exist, they are concatenated with commas.	Atlas 350 PCIe card, Atlas 850 series hardware products, and Atlas 950 SuperPoD use huawei.com/npu-Unhealthy as the parameter name.
huawei.com/Ascend910-Recovering	Marks the chips that are currently being recovered on the current node. When multiple chips exist, they are concatenated with commas.	Atlas 350 PCIe card, Atlas 850 series hardware products, and Atlas 950 SuperPoD use huawei.com/npu-Recovering as the parameter name.
huawei.com/Ascend910-Fault	Records specific fault information of the chip.	Array object. The object contains seven fields: fault_type, npu_name, large_model_fault_level, fault_level, fault_handling, fault_code, and fault_time_and_level_map. Atlas 350 PCIe card, Atlas 850 series hardware products, and Atlas 950 SuperPoD use huawei.com/npu-Fault as the parameter name.
-fault_type	Fault type.	CardUnhealthy: Chip fault CardNetworkUnhealthy: Chip network fault NodeUnhealthy: Node fault
-npu_name	Name of the faulty chip. It is empty for a node fault.	String
-large_model_fault_level -fault_level -fault_handling	Fault handling type. The value is empty for a node fault.	NotHandleFault: No handling RestartRequest: In inference scenarios, the inference request needs to be re-executed. In training scenarios, the training workload needs to be re-executed. RestartBusiness: The workload needs to be re-executed. FreeRestartNPU: Affects workload execution. The chip needs to be reset when it is idle. RestartNPU: Directly reset the chip and re-execute the workload. SeparateNPU: Isolate chip PreSeparateNPU: Pre-isolate chip. Whether to reschedule is determined based on the actual running status of the training task. [!NOTE] NOTE The large_model_fault_level, fault_level, and fault_handling parameters have the same function. It is recommended to use fault_handling. If an inference task subscribes to fault information, and a RestartRequest fault occurs on the inference card used by the task, and the fault duration does not exceed 60 seconds, task rescheduling is not performed. If the fault duration exceeds 60 seconds and the fault is not recovered, the chip is isolated and task rescheduling is performed.
-fault_code	Fault code, a string concatenated with commas.	For a detailed description of chip fault codes, see Chip Fault Code References.
-fault_time_and_level_map	Fault code, fault generation time, and fault handling level.	-
SuperPodID	SuperPod ID.	String
ServerIndex	The relative position of the current node within the SuperPod.	When the SuperPodID or ServerIndex value reported by the driver is 0xffffffff, the value of SuperPodID or ServerIndex is -1. The value of SuperPodID or ServerIndex is -2 in the following cases. The current device does not support querying SuperPod information. Failed to obtain SuperPod information due to a driver issue.
CheckCode	Check code.	-

The UnifiedBus device fault information reported by Ascend Device Plugin is shown in Table 2.

Table 2 SwitchInfoCfg

Name	Meaning	Description
FaultCode	List of UnifiedBus device fault codes for the current node.	Array object, containing fields such as EventType, AssembledFaultCode, PeerPortDevice, PeerPortId, SwitchChipId, SwitchPortId, Severity, Assertion, and AlarmRaisedTime.
-EventType	Alarm ID.	-
-AssembledFaultCode	Fault code.	-
-PeerPortDevice	Peer device type.	0: CPU 1: NPU 2: SW 0xFFFF: NA
-PeerPortId	Peer device ID.	-
-SwitchChipId	UnifiedBus device fault chip ID.	Numbering starts from 0.
-SwitchPortId	UnifiedBus device fault port ID.	Numbering starts from 0.
-Severity	Fault level.	0: Info 1: Minor 2: Major 3: Critical
-Assertion	Event type.	0: Fault recovery 1: Fault generation 2: Notification event
FaultLevel	Fault handling level of the current node.	Takes the highest fault level among all faults in FaultCode. Values include: NotHandle, SubHealthFault, Separate, and RestartRequest.
UpdateTime	Fault report refresh time.	-
NodeStatus	Health status of the current node.	Corresponds to FaultLevel values: NotHandle:Healthy, SubHealthFault:SubHealthy, Separate:UnHealthy, and RestartRequest:UnHealthy.
FaultTimeAndLevelMap	List of fault occurrence times and fault handling levels.	Array object, containing fields for fault code, UnifiedBus device fault chip ID, UnifiedBus device fault port ID, fault_time, and fault_level. The key is composed of the fault code, UnifiedBus device fault chip ID, and UnifiedBus device fault port ID, connected by underscores.
-fault_time	Fault occurrence time.	-
-fault_level	Fault handling level.	-

The manually intervened fault-level chip information reported by the ConfigMap of Ascend Device Plugin is shown in Table 3.

Table 3 ManuallySeparateNPU

Name	Meaning	Description
ManuallySeparateNPU	The chip is recorded in this key by the ConfigMap because multiple chip faults have triggered the frequency-based fault escalation policy.	Multiple chip names are separated by commas (,).

The fault policy escalation reasons reported by the ConfigMap of Ascend Device Plugin are shown in Table 4.

Table 4 UpgradeFaultReason

Name	Meaning	Description
UpgradeFaultReason	After a fault code is configured with frequency-based and duration-based policies, when fault escalation is triggered, this records the reason for the fault escalation and the escalation time.	JSON Map format, where the key is the chip name and the value is the reason that caused the chip fault escalation.
-fault_code	Fault code for chip fault escalation.	-
-fault_level	Fault level after escalation.	-
-upgrade_type	Fault escalation type.	Frequency-based escalation: FaultFrequency Duration-based escalation: FaultDuration Autofill escalation: FaultAutofill
-upgrade_time	Time point of fault escalation.	-

Note

When Ascend Device Plugin is upgraded from a version earlier than 26.0.0 to 26.0.0 or later, if an existing chip fault in the ConfigMap of Ascend Device Plugin is escalated to ManuallySeparateNPU, a reason for isolating that chip will be automatically filled. The -fault_code value will be AutofillFaultCode, and the -upgrade_type will be FaultAutofill.
The fault escalation reason is deleted along with fault downgrade, and the deletion event is recorded in the event events under the kube-system namespace of K8s.

For the description information in the ConfigMap of Ascend Device Plugin, see Table 5.

Table 5 Description

Name	Meaning	Description
Description	Description information.	The available chip information of the node in this ConfigMap is being phased out. By default, the available chips of the node are maintained by Volcano, and the information maintained in this ConfigMap does not take effect. If it needs to take effect, you can modify the Volcano configuration parameter "self-maintain-available-card" to false.

The NPU device fault information reported by Ascend Device Plugin is shown in Table 6. The object name is <device-plugin-pod-name>.<reporting-time><fault- chip-ID>, and the object type is Event.

Note

The following table only shows the field descriptions related to MindCluster services. For details about more fields, see Event core.

Table 6 NPU device fault information

Name	Meaning	Note
type	Event level.	Unique value: Warning
message	Event content, including node name, chip ID, fault generation or recovery type, fault code, and fault level information.	String
reason	Reason for event reporting.	Recovery: Fault recovery Occur: Fault generation Notice: One-time fault notification
action	Fault level.	String. For details, see Table 1.
source	Source of the fault.	Struct. Indicates the node where the fault occurred.
eventTime	Time when the fault occurred.	Timestamp
involvedObject	Object to which the fault is bound for display.	Struct. Points to the Pod name of the current Ascend Device Plugin through Kind, Namespace, and Name. After specification, the event can be viewed not only through the Event object query but also when viewing the details of the current Pod.
reportingComponent	Event controller.	Unique value: device-plugin
reportingInstance	Event reporting instance.	String. Takes the Pod name of the current Ascend Device Plugin.

deviceNameCustomization.json

deviceNameCustomization.json supports customizing device names. When building the Ascend Device Plugin image, place this file in the same directory as the binary package to change the resource type and resource name displayed by Ascend Device Plugin to custom names. Atlas 350 PCIe card, Atlas 850 series hardware products, and Atlas 950 SuperPoD do not currently support this feature.

Table 7 Custom device names supported by deviceNameCustomization.json

Name	Meaning	Note
ResourceType	Initial name of the device. Mandatory.	Only Ascend910, Ascend310, or Ascend310P is supported.
DevicePublicType	Type displayed externally for the device, for example, huawei.com/Ascend910. Mandatory.	Only the xxx.xxx/xxx format is supported. xxx can be uppercase and lowercase letters and digits, with a length range of 10 to 32 characters.
DevicePublicNamePre	Prefix of the device name displayed externally, for example, Ascend910-. For the actual displayed name, Ascend Device Plugin appends the physical ID of the chip after the prefix. Mandatory.	Can contain uppercase and lowercase letters, hyphens (-), and digits. Must start with an uppercase or lowercase letter. Length range: 2 to 16 characters.
PodConfigurationName	Details of the mounted chip information displayed in the Pod annotation. Mandatory when ResourceType is Ascend910.	Can contain uppercase and lowercase letters, hyphens (-), slashes (/), dots (.), and digits. Must start with an uppercase or lowercase letter and end with an uppercase or lowercase letter or digit. Length range: 10 to 63 characters.

Job Information

fault-config-job-name

Table 1 fault-config-job-name

Field	Description	Value	Remarks
fault-npus	Rank information of the faulty chip in a failed job.	String	-
checkCode	Check code.	String	-

reset-config-job-name

Table 2 reset-config-job-name

Field	Parameter	Meaning	Value	Remarks
reset.json	RankList	Chip list	-	-
	-RankId	Rank information used by the failed job	Integer	-
	-LogicId	Chip logic ID	32-bit integer	-
	-Status	Chip status	unrecovered: Not recovered recovered: Recovery succeeded failed: Recovery failed	-
	-Policy	Hot reset policy	empty: No fault ignore: Ignore the fault restart_request: Re-execute the current request restart: Re-execute the training task free_reset: When no job is running on the NPU, the device needs to be reset reset: The device needs to be reset isolate: The device needs to be isolated	-
	-InitialPolicy	Initial hot reset policy	empty: No fault ignore: Ignore the fault restart_request: Re-execute the current request restart: Re-execute the training job free_reset: When no job is running on the NPU, the device needs to be reset reset: The device needs to be reset isolate: The device needs to be isolated	-
	-ErrorCode	Decimal fault code	64-bit integer array	-
	-ErrorCodeHex	Hexadecimal fault code	String	-
	GracefulExit	Manage training processes	0 or 1 Value 1: Terminate all training processes Value 0: No action	-
	UpdateTime	ConfigMap update time	-	-
	RetryTime	Number of Pod rescheduling attempts	Integer	-
	FaultFlushing	Informs Elastic Agent whether a fault is currently being flushed	Value is true or false true: Indicates a fault is being flushed false: Indicates no fault is being flushed currently	Elastic Agent needs to wait until this field is false and the faulty RankList contains no faults for this node before starting the training process
	RestartFaultProcess	Informs Elastic Agent whether to restart only the faulty process on this node	Value is true or false true: Indicates not to exit Elastic Agent, only restart the faulty process on this node false: When this node has a faulty process, exit Elastic Agent	-
restartType	-	Type of reset.json update	podReschedule or hotReset	Value is podReschedule for single Pod rescheduling, and hotReset for hot recovery scenarios
checkCode	-	Check code	String	-

data-trace-job_name

Stores the on/off status of various tracing types for the current job. It is mounted to the compute node by Ascend Device Plugin for storage. After the training container mounts this file, TaskD reads it to trace various data.

Table 3 data-trace-job_name ConfigMap

Field	Meaning	Value	Type
Communication	Identifies the communication operator.	on/off	string
Step	Identifies the Step latency.	on/off	string
SaveCheckpoint	Identifies the SaveCheckpoint duration.	on/off	string
FP	Identifies the forward propagation data.	on/off	string
DataLoader	Identifies the DataLoader duration.	on/off	string

Note

This ConfigMap must be in the same namespace as the training job (data-trace-<job_name>), and include the label reset=true.
This ConfigMap is mounted by the Ascend Device Plugin to the /user/cluster-info/datatrace-config/namespace.data-trace-job_name/* folder on the training node, with the file name profilingSwitch.
If the user does not create this ConfigMap, ClusterD will attempt to automatically create it upon the first call to the gRPC interface ModifyTrainingDataTraceSwitch.
To use this feature, the user should mount the profilingSwitch file on the node into the /user/cluster-info/datatrace-config/ directory within the container using the hostPath method.
Currently, Step, SaveCheckpoint, FP, and DataLoader are enabled by default, and these four can only be enabled or disabled synchronously. When all five fields are off, all instrumentation is disabled; otherwise, the above four are enabled by default, and the communication operator is enabled or disabled based on its switch status.

steptime-dtpgroup

Stores the save path and start/stop switch for the job's iteration latency and grouping information. When starting a job, users can configure ConfigMap parameters through the CCAE management platform to determine whether the job has degraded.

Table 4 steptime-dtpgroup ConfigMap

First-level Parameter	Second-level Parameter	Meaning	Value	Remarks
data	PerfDumpPath	Path for saving iteration latency and grouping information.	String	-
-	PerfDumpConfig	Start/stop switch for iteration latency and grouping information.	String	-

Custom Chip Faults

Fault Levels in faultCode.json

The hierarchical handling based on different chip fault levels is introduced in resumable training. If you need to modify the fault level of a fault code, see (Optional) Configuring Chip Fault Levels.

After Ascend Device Plugin obtains chip fault codes from the driver, it classifies the faults into the following levels based on their impact on devices and workloads. For a detailed description, see Table 1.

Table 1 Fault level and handling polices

Fault Handling Policy	Description	Rescheduling Policy	Graceful Fault Tolerance
NotHandleFault	A fault that does not affect services and does not need to be handled.	Not handled for now	Not handled for now
RestartRequest	Affects service execution and requires re-execution of the service request.	Isolate the chip and reschedule the job Note: If an inference job subscribes to fault information, and a RestartRequest fault occurs on the inference card used by the job and the fault duration does not exceed 60 seconds, job rescheduling will not be performed. If the fault duration exceeds 60 seconds and the fault is not recovered, the chip will be isolated and job rescheduling will be performed.	Re-execute the inference request in inference scenarios, and re-execute the training workload in training scenarios
RestartBusiness	Affects service execution and requires re-execution of the service.		Re-execute the service
FreeRestartNPU	Affects service execution. The chip needs to be reset when it is idle.		Wait for the chip to become idle and then reset it
RestartNPU	Affects service execution. The chip needs to be reset immediately.		Immediately stop the training workload, reset the chip, and then re-execute the service
SeparateNPU	Unrecoverable. The chip needs to be isolated.		Isolate the chip and reschedule the job
PreSeparateNPU	Does not affect services for now, but jobs will no longer be scheduled to this chip subsequently.	Pre-isolate chip	Pre-isolate chip
SubHealthFault	Handled based on the value of the subHealthyStrategy parameter configured in the job YAML. For details, see Table 1 YAML parameter description.	When a sub-health fault occurs on a chip, it needs to be handled according to the policy in Configuring YAML. [!NOTE] Note: If a fault of another level occurs on the chip subsequently, the SubHealthFault handling policy does not affect the handling of faults at other levels.	Handle according to the policy

faultCustomization.json

If the user does not manually modify the faultCustomization.json file, Ascend Device Plugin performs fault handling according to the default configuration (default values) of faultCustomization.json.

Table 2 faultCustomization.json

First-level Parameter	Second-level Parameter	Description
GraceTolerance	-	Graceful tolerance related configuration. If GraceTolerance and its sub-parameters do not exist or exceed the value range, the default values are used.
-	WaitProcessReadCMTime	When using graceful tolerance mode, the time to wait for the management process to read the ConfigMap file, in seconds. Value range: 5 to 90. Default value is 30.
-	WaitDeviceResetTime	When using graceful tolerance mode, the maximum time to wait for the chip to restart, in seconds. Value range: 60 to 180. Default value is 150.
-	WaitFaultSelfHealingTime	When using graceful tolerance mode, the time to wait for RestartBusiness level fault recovery, in seconds. Value range: 1 to 30. Default value is 15.
FaultFrequency	-	Custom fault frequency. When the number of occurrences of a fault within the time window reaches the upper limit, the fault is handled according to the configured fault handling policy. If the value range of FaultFrequency or its sub-parameters is incorrect, this configuration is ignored. If the data format of FaultFrequency or its sub-parameters is incorrect, the default configuration is used.
-	EventId	Fault code ID. Only one FaultFrequency parameter is allowed for each fault code (EventId). If multiple are configured, only the first valid one takes effect.
-	TimeWindow	Time window, which counts the number of fault occurrences from the current time minus TimeWindow to the current time, in seconds. Value range: 60 to 864,000.
-	Times	The upper limit of occurrences for the same fault. Value range: 1 to 100. If the number of occurrences of this fault within the time window is greater than or equal to this value, it is handled and reported according to the policy defined in FaultHandling.
-	FaultHandling	The handling policy for the fault after the number of occurrences reaches the upper limit. Supports configuring fault handling policies at different levels. If ReleaseTimeWindow is configured, the policy can be automatically released when conditions are met. To support manual fault removal, configure the handling policy as ManuallySeparateNPU. PreSeparateNPU: This fault handling mode pre-isolates the chip and determines whether to reschedule based on the actual running status of the training task. ManuallySeparateNPU: When this policy is triggered, it directly reports to K8s that the chip is unhealthy and writes the chip name to device-info-cm. As long as the chip name is saved in this field, the chip remains isolated even if the fault recovers, until the maintenance personnel manually delete the chip name from this field, or the recovery duration exceeds ReleaseTimeWindow. This field only allows Ascend Device Plugin to add or modify it. Maintenance personnel can only delete chip names from this field. This policy is currently not supported in faultCode.json.
-	ReleaseTimeWindow	If the fault has recovered and no recurrence of the fault occurs for a duration exceeding ReleaseTimeWindow, the escalated fault handling policy is released. The value range of this parameter is 60 to the maximum value of uint32, in seconds. If this parameter is not configured, it means the policy will not be downgraded after escalation. Since ManuallySeparateNPU supports manual release, this parameter must be configured for all other handling policies except ManuallySeparateNPU to prevent the device from being permanently unusable after fault recovery.
FaultDuration	-	Custom fault timeout policy. When the duration of a fault reaches the configured upper limit, the fault is handled according to the specified fault handling policy. If the value range of FaultDuration or its sub-parameters is incorrect, this configuration is ignored. If the data format of FaultDuration or its sub-parameters is incorrect, the default configuration is used.
-	EventId	Fault ID. Only one FaultDuration parameter is allowed for each fault code (EventId). If multiple are configured, only the first valid one takes effect.
-	FaultTimeout	If the fault duration exceeds this value, the fault is handled according to the fault handling policy defined in FaultHandling, in seconds. Value range: 0 to 600. The default values are described as follows. For the parameter plane network fault with fault ID 81078603, the default value is 20. For the on-chip memory multi-bit fault with fault ID 80E01801, the default value is 30. For other faults, the default value is 0.
-	RecoverTimeout	If the fault recovery time exceeds this value, fault recovery is reported, in seconds. Value range: 0 to 86,400. The default values are described as follows. For the parameter plane network fault with fault ID 81078603, the default value is 60. Setting it to 0 is not recommended; it is recommended to be greater than the listWatchPeriod health status check cycle. For details about listWatchPeriod, see the "Ascend Device Plugin Startup Parameters" table in Ascend Device Plugin. For other faults, the default value is 0.
-	FaultHandling	The fault handling policy after the fault duration is exceeded. Supports configuring fault handling policies at different levels, and also supports configuring the PreSeparateNPU fault handling policy. The fault handling policy after the fault duration is exceeded is recommended to be higher than the fault handling policy of the fault itself; otherwise, the configuration does not take effect. Configuring the ManuallySeparateNPU policy is not supported, and the configuration does not take effect.

Note

If a fault code is configured with both a fault frequency (FaultFrequency) and a fault timeout policy (FaultDuration), and the number of timeouts for this fault code within the TimeWindow reaches the maximum number supported by the job, the most severe level among the following three will be used for handling. The three are: the fault handling policy of the fault itself, and the fault handling policies configured in FaultFrequency and FaultDuration.
If a fault code is configured with both a fault frequency and a fault timeout policy, the fault is considered to have occurred only after the fault timeout, and the frequency count increases by one. The fault is considered recovered only after the recovery exceeds RecoverTimeout. After recovery, the next count can only be accumulated upon another fault timeout.
The network fault with fault ID 81078603 only supports configuration of three fault handling policies: NotHandleFault, PreSeparateNPU, or SeparateNPU. If configured with other policies, the default configuration NotHandleFault will be used.
When Ascend Device Plugin is upgraded from a version earlier than 26.0.0 to version 26.0.0 or later, if the ConfigMap of Ascend Device Plugin already contains the ManuallySeparateNPU key-value, its degradation time window is the maximum ReleaseTimeWindow value in faultCustomization.json. If no fault code is configured with ReleaseTimeWindow, the existing ManuallySeparateNPU in the ConfigMap will not be degraded.
After modifying/deleting policies in mindx-dl-fault-config, the upgraded fault handling policies will also be updated with the configuration changes. However, after the ManuallySeparateNPU policy is upgraded, deleting the fault handling policy configuration item will not remove it; the removal method for ManuallySeparateNPU must be followed. Additionally, when other fault level configurations are updated to ManuallySeparateNPU, the configuration will not take effect or will not meet expectations.

Custom UnifiedBus Device Faults

Resumable training performs hierarchical handling based on different levels of UnifiedBus bus device faults. If users need to modify the fault level of a fault code, see (Optional) Configuring UnifiedBus Device Fault Levels.

After Ascend Device Plugin obtains fault codes from the driver, it classifies faults into the following five levels based on their impact on devices and workloads, and performs corresponding rescheduling handling. For detailed description, see Table 1.

Table 1 Fault levels and handling policies

Fault Type	Description	Rescheduling Policy
NotHandleFault	Does not affect workloads temporarily and can recover automatically. No handling required.	Not handled for now.
SubHealthFault	Affects workload running performance. The cause of sub-health fault needs to be investigated.	When a sub-health fault occurs, it needs to be handled according to the sub-health policy specified by the subHealthyStrategy parameter in Table 1 YAML parameter description.
RestartRequestFault	Workload running fails. The workload request needs to be re-executed.	Stop the current training job, isolate the node, and reschedule the job.
ResetFault	Workload running fails.	Stop the current training job, isolate the node, and reschedule the job.
SeparateFault	Workload running fails. The device or board needs to be replaced.	Stop the current training job, isolate the node, and reschedule the job.