Obtaining Cluster Scheduling Component Information
The VolcanoJob API is provided by the open-source component Volcano. MindCluster has modified the Annotations field of the VolcanoJob API, as shown in Table 1. Other APIs remain unchanged. For details about open-source Volcano, see the Volcano open-source community.
Table 1 Annotations
Parameter
Description
Value
distributed
Written and used by Resilience Controller to mark whether a job is a distributed task.
true
The following describes the interfaces exposed by the volcano-scheduler and volcano-controller component Pods (defined by the open-source components themselves).
Table 2 List of interfaces exposed by MindCluster Volcano
Access Method
Protocol
Method
Description
Component
http://podIP:11251/healthz
http
Get
Health check port
volcano-controller
http://podIP:11251/healthz
http
Get
Health check port
volcano-scheduler
http://volcano-scheduler-serviceIP:8080/metrics
http
Get
Prometheus metrics collection port
volcano-scheduler
Note
To ensure normal access to the Volcano health check port and the Prometheus metrics collection port, set the --enable-healthz and --enable-metrics parameters to true in the YAML file when installing Volcano. For details about how to modify these parameters, see "Step 7" in Installing Volcano.
The HUAWEI CLOUD CCI service provides more detailed VolcanoJob instructions. For details, see the "Creating a VolcanoJob" section in Cloud Container Instance.
PodGroup
Table 1 PodGroup labels used by cluster scheduling components
Name
Description
Value
Component
ring-controller.atlas
Identifies Atlas Pods
ascend-npu
ascend-910
ascend-{xxx}b
Ascend Device Plugin, Ascend Operator, Volcano
fault-scheduling
Rescheduling upon job faults
grace, force, off
Volcano, Resilience Controller
elastic-scheduling
Job elastic scheduling
on
Volcano, Resilience Controller
fault-retry-times
Number of times a job can be rescheduled when a service plane fault occurs
0-100
Volcano, Ascend Operator
tor-affinity
Switch affinity policy
normal-schema
large-model-schema
null
Volcano
npu-310-strategy
Marks the scheduling policy for inference servers (with Atlas 300I inference cards)
card
chip
Volcano
pod-rescheduling
Enable Pod-level rescheduling.
on: Enable Pod-level rescheduling
Other values or if this field is not used: Disable Pod-level rescheduling
Volcano
process-recover-enable
Enable process-level rescheduling.
on: Enable process-level rescheduling
Other values or if this field is not used: Disable process-level rescheduling
Volcano
subHealthyStrategy
Sub-health handling policy.
ignore: Ignore the sub-healthy node. Subsequent jobs will not prioritize this node in affinity scheduling.
graceExit: Do not use the sub-healthy node. Save the checkpoint file and then perform rescheduling. Subsequent jobs will not be scheduled to this node.
forceExit: Do not use the sub-healthy node. Exit the jobs without saving and perform rescheduling. Subsequent jobs will not be scheduled to this node.
hotSwitch: Perform a hot switching. After the backup Pod is started, pause the training jobs and use a new node to restart training.
Volcano
huawei.com/scheduler.softShareDev.aicoreQuota
Percentage of AICore requested by the soft partitioning job.
[1, 100]
Volcano
huawei.com/scheduler.softShareDev.hbmQuota
Amount of high-bandwidth memory requested by the soft partitioning job.
[1, maxHBM]
maxHBM is the HBM value in HBM-Usage(MB) queried using the npu-smi info command.
Volcano
huawei.com/scheduler.softShareDev.policy
Policy for the soft partitioning job.
fixed-share
elastic
best-effort
Volcano
Table 2 PodGroup annotations used by cluster scheduling components
Name
Description
Value
Component
sp-block
Specifies the number of chips in a logical SuperPoD.
idlest: Logical SuperPoDs are scheduled to more idle physical SuperPoDs.
Non-idlest: Logical SuperPoDs preferentially fill up physical SuperPoDs.
Volcano
huawei.com/schedule_minAvailable
Minimum number of replicas required for a job to be scheduled.
Integer
Volcano
huawei.com/recover_policy_path
Job rescheduling policy.
pod: Only supports Pod-level rescheduling, and will not escalate to the Job level. (When using vcjob, you need to configure this policy: policies: -event:PodFailed -action:RestartTask)
Volcano
huawei.com/schedule_enable_dequeue
Enables or disables the job dequeue function (transitioning from Inqueue to Pending state). Manual configuration is required.
"on": Enable
Other values: Disable
If not configured, it is disabled by default.
Volcano
huawei.com/schedule_dequeue_frequency
Records the number of times a job is dequeued. Automatically updated by Volcano.
This value increments by 1 each time the job is dequeued.
This value is deleted when the job is not in the Inqueue or Pending state.
Volcano
huawei.com/schedule_enqueue_time
Records the time when a job is enqueued (transitioning from Pending to Inqueue state). Automatically updated by Volcano.
Millisecond-level timestamp.
If a job has been enqueued for more than 5 minutes and the dequeue function is enabled, when other jobs need to be enqueued, this job will be dequeued to release resources so that other jobs can be enqueued.
This value is deleted when the job is not in the Inqueue state.
Volcano
huawei.com/affinity-config
Configures the affinity hierarchy for multi-level scheduling of jobs.
level1=x,level2=y,...
Where x, y... are the subjob sizes for the corresponding network hierarchy.
The format must be a concatenation of strings in the format leveli=ni, separated by commas. Here, i is the network hierarchy sequence number, and ni is the number of replicas for the subjob at that network hierarchy. For example, for a job with a total of 8 replicas, "level1=2,level2=4" means that every 2 Pods in the job are assigned to nodes with the same level1 label, and every 4 Pods are assigned to nodes with the same level2 label.
The network hierarchy configuration must meet the following requirements:
When the job hierarchy has more than 1 level, the value of level n must be an integer multiple of n-1.
The total number of job replicas must be an integer multiple of all hierarchy levels.
The job hierarchy configuration must start from level1 and be consecutive in ascending order.
Volcano
Table 3 huawei.com/schedule_policy configurations
Configuration
Description
chip4-node8
1 node with 8 chips, every 4 chips form an interconnect ring. For example, Atlas 800 training server (model 9000)/Atlas 800 training server (model 9010)/Atlas 350 PCIe card has 8 chips total, every 4 chips are connected via UB mezzanine.
chip1-node2
1 node with 2 chips. For example, the Atlas 300T training card supports up to 1 chip, and 1 node supports up to 2 chips.
chip4-node4
1 node with 4 chips, forming 1 interconnect ring, for example, Atlas 800 training server (models 9000, 9010) half-configuration chip scenario.
chip8-node8
1 node with 8 chips. All 8 chips form 1 interconnect ring. For example, Atlas 800T A2 training server/Atlas 850 series hardware products.
chip8-node16
1 node with 16 chips, every 8 chips on 1 interconnect ring. For example, Atlas 200T A2 Box16 heterogeneous subrack.
chip2-node8
1 node with 8 chips, every 2 chips on 1 interconnect ring.
chip2-node16
1 node with 16 chips, every 2 chips on 1 interconnect ring. For example, Atlas 800T A3 SuperPoD.
chip2-node8-sp
1 node with 8 chips, every 2 chips on 1 interconnect ring. Multiple servers form a SuperPoD, for example, Atlas 9000 A3 SuperPoD cluster computing system.
chip2-node16-sp
1 node with 16 chips, every 2 chip on 1 interconnect ring. Multiple servers form a SuperPoD, for example, Atlas 900 A3 SuperPoD.
chip4-node16
1 node with 16 chips, every 4 chips on 1 interconnect ring. For example, Atlas 350 PCIe card with 16 chips total, every 4 chips connected via UB mezzanine.
chip1-node8
1 node with 8 chips, no interconnect between chips. For example, Atlas 350 PCIe card with 8 chips total, no interconnect between chips.
chip1-node16
1 node with 16 chips, no interconnect between chips. For example, Atlas 350 PCIe card with 16 chips total, no interconnect between chips.
chip8-node8-sp
1 node with 8 chips, all 8 chips on 1 interconnect ring. Multiple servers form a SuperPoD, for example, Atlas 850 SuperPoD.
chip8-node8-ra64-sp
1 node with 8 chips, all 8 chips on 1 interconnect ring, 64 nodes form a computing cabinet. Multiple cabinets form a SuperPoD, for example, Atlas 950 SuperPoD.
chip1-softShareDev
Dedicated scheduling policy for soft partition-based virtualization.
multilevel
Used in multi-level scheduling scenarios. For details on how to use multi-level scheduling, see Multi-level Scheduling.
Pod
Table 1 Pod labels used by cluster scheduling components
Name
Description
Value
Component
ring-controller.atlas
Identifies the Atlas Pod
ascend-910
ascend-{xxx}b
huawei.com/npu
Ascend Device Plugin, Ascend Operator, Volcano
fault-scheduling
Switch for job rescheduling upon faults
grace, force, off
Volcano, Resilience Controller
elastic-scheduling
Switch for job elastic scheduling
on
Resilience Controller, Volcano
fault-retry-times
Number of times a job can be rescheduled when a service-plane fault occurs
0-100
Volcano, Ascend Operator
tor-affinity
Switch affinity policy
normal-schema
large-model-schema
null
Volcano
npu-310-strategy
Marks the scheduling policy for the inference server (with Atlas 300I inference card)
card
chip
Volcano
pod-rescheduling
Whether to enable Pod-level rescheduling.
on: Enable Pod-level rescheduling
Other values or when this field is not used: Disable Pod-level rescheduling
Volcano
process-recover-enable
Whether to enable process-level rescheduling.
on: Enable process-level rescheduling
Other values or when this field is not used: Disable process-level rescheduling
Volcano
subHealthyStrategy
Sub-health handling strategy
ignore: Ignore the sub-healthy node, and subsequent jobs will not prioritize this node in affinity scheduling.
graceExit: Do not use the sub-healthy node, save the dying gasp CKPT file, and then perform rescheduling. Subsequent jobs will not be scheduled to this node.
forceExit: Do not use the sub-healthy node, exit the job directly without saving, and perform rescheduling. Subsequent jobs will not be scheduled to this node.
hotSwitch: Perform a hot switching, start the backup Pod, pause the training job, and use a new node to restart the training.
Volcano
Table 2 Pod annotations used by cluster scheduling components
Name
Description
Value
Component
sp-block
Specifies the number of chips in a logical SuperPoD.
idlest: The logical SuperPoD is scheduled to a more idle physical SuperPoD.
Non-idlest: The logical SuperPoD preferentially fills up a physical SuperPoD.
Volcano
huawei.com/schedule_minAvailable
The minimum number of replicas required for the job to be scheduled.
Integer
Volcano
huawei.com/recover_policy_path
Job rescheduling policy.
pod: Only Pod-level rescheduling is supported, and it will not be escalated to the Job level. (When using vcjob, you need to configure this policy: policies: -event:PodFailed -action:RestartTask)
Volcano
huawei.com/affinity-config
Configures the affinity hierarchy for multi-level scheduling of the job.
level1=x,level2=y,...
Where x, y... are the subjob sizes at the corresponding network hierarchy.
The format must be a concatenation of strings in the format leveli=ni, separated by commas. Here, i is the network hierarchy sequence number, and ni is the number of replicas for the subjob at that network hierarchy. For example, for a job with a total of 8 replicas, "level1=2,level2=4" means that every 2 Pods in the job are assigned to nodes with the same level1 label, and every 4 Pods are assigned to nodes with the same level2 label.
The network hierarchy configuration must meet the following requirements:
When the job hierarchy has more than 1 level, the value of level n must be an integer multiple of n-1.
The total number of job replicas must be an integer multiple of all hierarchy levels.
The job hierarchy configuration must start from level1 and be consecutive from small to large.
Volcano
huawei.com/skip-ascend-plugin
Allows some special jobs (such as jobs that do not require NPU resources) to bypass the default check logic of Ascend-for-volcano.
Not set or set to "enabled"
Volcano
Table 3 PodGroup Status.Conditions used by cluster scheduling components
Type
Status
Reason
Message
Unschedulable
True
JobEnqueueFailed
The job failed to enqueue due to insufficient cluster NPU resources.
Unschedulable
True
JobValidateFailed
job validation failed, typically due to an insufficient number of pods, non-compliant resource requests, or an unsupported job type.
Set of chips with network faults on the faulty node
String slice
-
- NodeDEnable
Whether the node status detection switch is enabled
True
False
-
- NodeHealthState
Node health status
String
-
FaultDeviceList
-
-
-
- fault_type
Fault type object
CardUnhealthy: Chip fault
CardNetworkUnhealthy: Chip network fault
NodeUnhealthy: Node Failure
-
- npu_name
Name of the faulty chip. Empty in case of a node failure.
String
-
- fault_level
Fault handling type. Empty in case of a node failure.
NotHandleFault: No handling
RestartRequest: Re-execute inference requests for inference scenarios; re-execute training services for training scenarios
RestartBusiness: Re-execute the service
FreeRestartNPU: Affects service execution. Reset the chip when it becomes idle.
RestartNPU: Directly reset the chip and re-execute the service
SeparateNPU: Isolate the chip
PreSeparateNPU: Pre-isolate the chip. Determine whether to reschedule based on the actual running status of the training task.
The fault_level, fault_handling, and large_model_fault_level parameters have the same functionality. It is recommended to use fault_handling.
If an inference task subscribes to fault information and a RestartRequest fault occurs on the inference card used by the task, and the fault duration does not exceed 60 seconds, task rescheduling will not be performed. If the fault duration exceeds 60 seconds and is not recovered, the chip will be isolated and task rescheduling will be performed.
- fault_handling
- large_model_fault_level
- fault_code
Fault code. A string concatenated by commas.
String
Disconnected: Chip network disconnection fault.
heartbeatTimeOut: Node status loss fault
remain-retry-times
Remaining reschedulable information for the job
-
-
- UUID
Task UID
String
-
- Times
Remaining reschedulable times for the job
Integer
-
reset-config-<job-name>
The MindCluster cluster scheduling component writes information such as device and training task status into the reset-config-<job-name> ConfigMap through K8s and maps it into the container. The Elastic Agent reads it and performs corresponding fault detection and processing.
Table 3 reset-config-<job-name>
Field Name
Name
Description
Value
Remarks
reset.json
RankList
Chip list
-
-
RankId
Rank information used by the faulty task
Integer
-
LogicId
Chip logic ID
32-bit integer
-
Status
Chip status
unrecovered: Not recovered
recovered: Recovery succeeded
failed: Recovery failed
-
Policy
Hot reset policy
empty: No fault
ignore: Ignore the fault
restart_request: Re-execute the current request
restart: Re-execute the training task
free_reset: The device needs to be restarted when there is no task on the NPU
reset: The device needs to be restarted
isolate: The device needs to be isolated
-
InitialPolicy
Initial hot reset policy
empty: No fault
ignore: Ignore the fault
restart_request: Re-execute the current request
restart: Re-execute the training task
free_reset: The device needs to be restarted when there is no task on the NPU
reset: The device needs to be restarted
isolate: The device needs to be isolated
-
ErrorCode
Decimal fault code
64-bit integer array
-
GracefulExit
Manages training processes
0 or 1
Value 1: Terminate all training processes
Value 0: No action
-
FaultFlushing
Informs Elastic Agent whether a fault is currently being flushed
Value is true or false
true: Indicates that a fault is being flushed
false: Indicates that no fault is being flushed
Elastic Agent needs to wait until this field is false and the fault RankList does not contain a fault for this node before starting the training process
RestartFaultProcess
Informs Elastic Agent whether to restart only the faulty process on this node
Value is true or false
true: When a fault occurs on this node, restart only the faulty process on this node
false: When a fault occurs on this node, exit all processes on this node and exit Elastic Agent
This field takes effect only when the fault RankList contains a fault for this node
ErrorCodeHex
Hexadecimal fault code
String
-
restartType
-
Type of reset.json update
podReschedule or hotReset
The value is podReschedule for single Pod rescheduling, and hotReset for hot recovery scenarios
mindx-dl/job-reschedule-reason
This ConfigMap is used to record the historical information of job rescheduling. By default, it saves the ten most recent rescheduling records of a job. When the ConfigMap content exceeds 950 KB, the earliest record of each job will be deleted in sequence.
Table 4 Job field description
Field
Name
Description
Value
Remarks
Job namespace/Job name
-
Marks the name of the job that performs rescheduling.
String
-
JobID
Job ID
String
-
TotalRescheduleTimes
The total number of rescheduling times recorded for this job in the current lifecycle of Volcano
Integer
-
RescheduleRecords
Records the specific information of the rescheduling of this task.
-
-
Table 5 RescheduleRecords description
Field Name
Name
Description
Value
Remarks
RescheduleRecords
LogFileFormatTime
Rescheduling time recorded in Volcano log format
String
-
RescheduleTimeStamp
Timestamp when rescheduling occurred
String
-
ReasonOfTask
Records the specific information of this rescheduling.
-
-
Table 6 ReasonOfTask Description
Field Name
Name
Description
Value
Remarks
ReasonOfTask
RescheduleReason
Reason for rescheduling
String
-
PodName
The pod that first triggered this rescheduling
String
-
NodeName
Node name
String
The node that first triggered this rescheduling.
NodeRankIndex
The rank of the node that first triggered this rescheduling in the training
String
-
Parameter Plane Network Topology
basic-tor-node-cm
Table 1 basic-tor-node-cm
Name
Description
Value
Remarks
version
Version of basic-tor-node-cm
1.0
-
tor_count
Number of nodes under the switch in the cluster
Integer
-
server_list
Collection of cluster nodes grouped by switch
-
-
- tor_id
Sequence number of the switch
Integer
-
- tor_ip
IP address of the switch
String
-
server
Node information under the switch
-
-
- server_ip
IP address of the node
String
-
- npu_count
Number of NPU chips on the node
Integer
-
- slice_id
Sequence number of the node under the switch
Integer
-
Volcano Configuration
volcano-scheduler-configmap
Table 1 Field description of volcano-scheduler-configmap
Name
Description
Value
Remarks
actions
Actions used in the scheduling process
String
ascend-volcano-plugin uses three scheduling actions: enqueue, allocate, and backfill
plugins
Set of plugins used in the scheduling process
-
-
- name
Name of the plugin in use
String
ascend-volcano-plugin uses
the following scheduling plugins: priority, gang, conformance, volcano-npu_{version}_linux-{arch}, drf, predicates, proportion, nodeorder, and binpack
configurations
Configuration information for scheduler initialization
