Environment Variable Description

Environment Variables Used by MindCluster Components

The environment variables used by MindCluster components are described in Table 1.

Table 1 Environment variables

Environment Variable Name	Source	Required	Value	Description
POD_IP	Written in the YAML of the deployment component	Yes	IP of the Pod where the current container resides	Used by ClusterD and TaskD to start the gRPC service
POD_UID	Written in the YAML of the deployment component	No	UID of the Pod where the current container resides	Used to parse the server_id field of the RankTable file
ASCEND_DOCKER_RUNTIME	Written by Ascend Docker Runtime during container creation	No	"true"	Used by Ascend Device Plugin to determine whether the default container runtime on the current node is Ascend Docker Runtime
HOSTNAME	Written by K8s during container creation	Yes	Name of the pod where the current container resides	Used by Ascend Device Plugin to obtain the current pod name
NODE_NAME	Written in the YAML of the deployment component	Yes	Name of the node where the current container resides	Used by Ascend Device Plugin, NodeD, and ClusterD to obtain the current node name
LD_LIBRARY_PATH	Written in the Dockerfile	Yes	File path	Used by Ascend Device Plugin and NPU Exporter to initialize DCMI
BATCH_BIND_NUM	-	No	Numeric string	Specifies the number of pods for Volcano to bind in batch
MULTI_SCHEDULER_ENABLE	-	No	"true" or "false"	Specifies whether Volcano is used in a multi-scheduler scenario
SCHEDULER_POD_NAME	-	No	String	Specifies the Volcano scheduler pod name
SCHEDULER_NUM	-	No	Numeric string	Specifies the number of Volcano schedulers
PANIC_ON_ERROR	-	No	"true" or "false"	Specifies whether the Volcano scheduler needs to panic when an error occurs
KUBECONFIG	-	No	File path	Specifies the kubeconfig path for Volcano to connect to the K8s api-server
HOME	Written by K8s during container creation	Yes	Folder path	Specifies the current user home path obtained by Volcano
DEBUG_SOCKET_DIR	-	No	Socket file path	Specifies the socket path that Volcano listens on
HCCL_CONNECT_TIMEOUT	Written in the training script	No	HCCL link establishment timeout	Indicates the link establishment timeout
TTP_PORT	Written in the YAML of the deployment component	Yes	Communication port used by MindIO TTP	Used to start MindIO Controller
SSH_CLIENT	Environment variable set by the SSH server, containing information about the client connection	Yes	Information about the current client connection	Records this information in the operation log when installing Ascend Docker Runtime
TASKD_LOG_PATH	-	No	String	Indicates the disk path for TaskD running logs
MINDX_SERVER_IP	Written by Ascend Operator during container creation	Yes	String	Indicates the IP address for communication between a job and ClusterD, which is also the svc IP of clusterd-grpc-svc
MINDX_SERVER_DOMAIN	Written by Ascend Operator during container creation	Yes	String	Indicates the domain name for communication between a job and ClusterD. Default Value is "clusterd-grpc-svc.mindx-dl.svc.cluster.local"
MINDX_TASK_ID	Written by Ascend Operator during container creation	No	For MindIE inference jobs, the value is the value of the jobID field under the label field in an acjob	Required for Elastic Agent/TaskD to register the gRPC service with ClusterD and for the TaskD profiling feature to save logs
GROUP_BASE_DIR	Written in the job startup script	No	Folder path	Indicates the parallel domain information export path of TaskD
MINDIO_WAIT_MINDX_TIME	Written in the job YAML	No	Numeric string, with a value range of [1, 3600]	Timeout for waiting for faulty pod scheduling when process-level rescheduling is not enabled and elastic training is enabled
RAS_NET_ROOT_PATH	User configuration	No	Root path of the shared directory between ClusterD and NodeD	In the slow network diagnosis scenario, ClusterD and NodeD interact through shared storage. For details, see Slow Network Diagnosis
REPLICA_TYPE	Written by Ascend Operator during container creation	Yes	Master, Scheduler, Chief, or Worker	Pod replica type

Ascend Operator Environment Variables

Ascend Operator provides corresponding environment variables for distributed training jobs (acjob) of different AI frameworks. For details, see the table below

Table 2 Training environment variables injected by Ascend Operator

Framework	Environment Variable	Function	Value	Description
PyTorch	MASTER_ADDR	IP address for communicating with the master node	Valid IP address in the format of string; must be in standard IPv4 or IPv6 format	clusterIP of the service corresponding to the master pod.
	MASTER_PORT	Port for communicating with the master node	String or number, with a value range of 0 to 65520	The value of the ascendjob-port field in the svc corresponding to the master pod. The default value is 2222.
	WORLD_SIZE	Total number of NPUs used by the job	Integer greater than 0	Total number of NPUs used by the job. For example, if there are 64 NPUs, the value is 64.
	RANK	Node rank of the pod on this node	Integer greater than or equal to 0	The value is 0 for the master node, and the value increases one by one for worker nodes.
	LOCAL_WORLD_SIZE	Number of NPUs used per node Pod	Integer greater than or equal to 0	For example, if a Pod uses 4 NPUs, configure it as 4.
	LOCAL_RANK	List of logical IDs of NPUs used per node Pod	String	Configured based on the number of NPUs used by the Pod, starting from 0. For example, if a Pod uses 4 NPUs, the configuration is {0,1,2,3}.
PyTorch, MindSpore	HostNetwork	Value of the hostNetwork field in the job YAML.	true: Creates a Pod using HostIP. false: Creates a Pod without using HostIP.	When the cluster is large (number of nodes > 1000), it is recommended to create Pods using HostIP.
	MINDX_SERVER_IP	IP address for communication between the job and ClusterD, which is also the svc ip of clusterd-grpc-svc.	A valid IP address. The format is a string and must be in standard IPv4 or IPv6 format.	-
	HCCL_LOGIC_SUPERPOD_ID	Chips with the same ID communicate using the UnifiedBus network, while chips with different IDs communicate using the RoCE network.	Integer greater than or equal to 0	HCCL uses this environment variable for dynamic networking to restrict the network communication mode between chips. This environment variable is only supported under the following conditions: Hardware: Atlas 900 A3 SuperPoD. Software: MindCluster 7.0.RC1 or later, CANN 8.0.0 or later.
	MINDX_TASK_ID	Elastic Agent/TaskD needs to provide the MINDX_TASK_ID information when registering the gRPC service with ClusterD. For the MindIE inference jobs, the value is the value of the jobID field under the label field in an acjob.	String	Job UID
	APP_TYPE	The value is the value of the app field under the label field in an acjob.	String	-
	REPLICA_TYPE	Pod replica type.	String. The value is Master, Scheduler, Chief, or Worker.	-
MindSpore	NPU_POD	Marks whether the current pod has a chip mounted.	true: The current pod has a chip mounted. false: The current pod does not have a chip mounted.	-
	MS_SERVER_NUM	Specifies the number of processes with the role MS_PSERVER.	0	The PS mode is not currently supported. Set this to a fixed value of 0. For detailed information about MS_PSERVER and the PS mode, see the relevant MindSpore documentation.
	MS_WORKER_NUM	Total number of NPUs used by the job	Integer greater than 0	Total number of NPUs used by the job. For example, if a job uses 64 NPUs, the value is 64.
	MS_LOCAL_WORKER	Number of NPUs used per node Pod	Integer greater than 0	For example, if a Pod uses 4 NPUs, configure it as 4.
	MS_SCHED_HOST	IP address of the Scheduler	Valid IP address, in the format of string; must be in standard IPv4 or IPv6 format.	In the Scheduler Pod, set to podIP. In the Worker Pod, set to the clusterIP of the Scheduler Pod's corresponding svc.
	MS_SCHED_PORT	Port for communicating with the Scheduler	Port number in the range of 1024 to 65535.	Value of the ascendjob-port field in the corresponding Scheduler Pod's svc. The default value is 2222.
	MS_ROLE	Specifies the role of this process.	MS_SCHED: Scheduler process. Only one Scheduler is started per training job, responsible for networking, container recovery, etc., and does not execute training code.. MS_WORKER: Worker process. Distributed training processes are generally set to this role.	The Worker process registers with the Scheduler process to complete networking.
	MS_NODE_RANK	Node rank of the Pod on this node	Integer greater than or equal to 0	0 for the Scheduler Pod. When the Scheduler mounts chips, Worker Pods start incrementing from 1. When the Scheduler does not mount chips, Worker Pods start incrementing from 0.
PyTorch, MindSpore	PROCESS_RECOVER	Master switch for process-level rescheduling, process-level online recovery, and elastic training.	on: Enable this feature. off: Disable this feature.	Inject this environment variable in process-level rescheduling, process-level online recovery, process-level in-place recovery, and elastic training scenarios.
PyTorch	HIGH_AVAILABILITY	Switch for the MindSpeed-LLM process-level recovery feature.	Available recovery strategies. retry: Process-level online recovery recover: Process-level rescheduling dump: Save dying gasps elastic-training: Elastic training
PyTorch, MindSpore	ELASTIC_PROCESS_RECOVER_ENABLE	Elastic Agent-side switch for process-level rescheduling, process-level online recovery, and dying gasp checkpoint features.	Value is 1: Enable. Value is other values: Disable. When this variable feature is, the related features on the MindIO side must also be disabled.	Inject this environment variable in process-level rescheduling, process-level online recovery, and process-level in-place recovery scenarios.
PyTorch, MindSpore	ENABLE_RESTART_FAULT_PROCESS	Process-level/in-place recovery switch for Elastic Agent and TaskD.	on: Enable this feature. Other values: Disable this feature. Note: Under the PyTorch framework, this feature is provided by Elastic Agent/TaskD. Under the MindSpore framework, this feature is provided by TaskD.
MindSpore	MINDIO_FOR_MINDSPORE	Switch for MindIO to start MindSpore.	Value is 1: Enable the switch for MindIO to start MindSpore. Value is not 1: Disable the switch for MindIO to start MindSpore.
MindSpore	MS_ENABLE_TFT	MindSpore process-level recovery switch.	{TTP:1,UCE:1,ARF:1,HCCE:1,RSC:1} # String type, respectively enable dying gasp, process-level online recovery for on-chip memory faults, process-level rescheduling, process-level online recovery for network faults, and pod-level rescheduling Note: For detailed descriptions of the above fields, see the "MS_ENABLE_TFT" environment variable in the MindSpore documentation.

Ascend Docker Runtime Environment Variable

Ascend Docker Runtime injects the environment variable below into the container.

Environment Variable	Function	Value	Description
ASCEND_DOCKER_RUNTIME	Identifies whether Ascend Docker Runtime is installed in the current environment.	True	This environment variable does not exist when Ascend Docker Runtime is not installed.

Ascend Device Plugin Environment Variables

Ascend Device Plugin injects corresponding environment variables into the container. See the following table for descriptions of these environment variables.

Table 3 Environment variables injected by Ascend Device Plugin into the container

Environment Variable	Function	Value	Description
ASCEND_VISIBLE_DEVICES	If a job requires NPU devices, ASCEND_VISIBLE_DEVICES must be used to specify the NPU devices to be mounted into the container; otherwise, NPU device mounting fails. When specifying devices by device index, both individual and range specifications are supported, and they can be used together. When specifying devices by chip name, multiple chip names of the same type can be specified simultaneously.	Mounting physical chips (NPUs) ASCEND_VISIBLE_DEVICES=0 indicates that NPU device 0 (/dev/davinci0) is mounted into the container. ASCEND_VISIBLE_DEVICES=1,3 indicates that NPU devices 1 and 3 are mounted into the container. Mounting virtual chips (vNPUs) Static virtualization: The usage is the same as for physical chips; simply replace the physical chip ID with the virtual chip ID (vNPU ID). Dynamic virtualization: ASCEND_VISIBLE_DEVICES=0 indicates that a certain number of AICores are allocated from NPU device 0.	-
ASCEND_ALLOW_LINK	Whether to allow soft links in mounted files or directories. This parameter must be specified in Atlas 500 A2 Intelligent Station, Atlas 200I A2 Accelerator Module, and Atlas 200I DK A2 scenarios.	ASCEND_ALLOW_LINK=True indicates that mounting driver files with soft links is allowed in Atlas 500 A2 Intelligent Station, Atlas 200I A2 Accelerator Module, and Atlas 200I DK A2 scenarios. ASCEND_ALLOW_LINK=False or if this parameter is not specified, Ascend Docker Runtime cannot be used on Atlas 500 A2 Intelligent Station, Atlas 200I A2 Accelerator Module, and Atlas 200I DK A2.	-
ASCEND_RUNTIME_OPTIONS	Imposes restrictions on the chip IDs specified in the ASCEND_VISIBLE_DEVICES parameter: NODRV: Indicates that driver-related directories are not mounted. VIRTUAL: Indicates that the mounted chip is a virtual chip. NODRV,VIRTUAL: Indicates that the mounted chip is a virtual chip and driver-related directories are not mounted.	ASCEND_RUNTIME_OPTIONS=NODRV ASCEND_RUNTIME_OPTIONS=VIRTUAL ASCEND_RUNTIME_OPTIONS=NODRV,VIRTUAL	-
WORLD_SIZE	Total number of NPUs used by a job	Integer greater than or equal to 0	Written only in dynamic vNPU scheduling scenarios
LOCAL_WORLD_SIZE	Number of NPUs used by each node's Pod	Integer greater than or equal to 0	Written only in dynamic vNPU scheduling scenarios
LOCAL_RANK	List of logical IDs of NPUs used by each node's Pod	String	Written only in dynamic vNPU scheduling scenarios. The value starts from 0. For example, if a Pod uses 4 NPUs, the configuration is {0,1,2,3}.
CM_WORKER_SIZE	Total number of NPUs used by a job	Integer greater than or equal to 0	Written only in dynamic vNPU scheduling scenarios
CM_LOCAL_WORKER	Number of NPUs used by each node's Pod	Integer greater than or equal to 0	Written only in dynamic vNPU scheduling scenarios
MS_WORKER_NUM	Total number of NPUs used by a job	Integer greater than or equal to 0	Written only in dynamic vNPU scheduling scenarios
MS_LOCAL_WORKER	Number of NPUs used by each node's Pod	Integer greater than or equal to 0	Written only in dynamic vNPU scheduling scenarios
PERF_DUMP_PATH	Path for saving iteration latency and grouping information	String	Written only in slow node detection scenarios
PERF_DUMP_CONFIG	Start/stop switch for iteration latency and grouping information	String	Written only in slow node detection scenarios
KUBELET_PORT	Specifies the default port number of kubelet on the current node (if the user has not customized the kubelet port, no configuration is required).	Integer from 0 to 65535	If the user modifies the default kubelet port, the value of this environment variable must be set to the custom port number. If the user has not modified the default kubelet port, this environment variable is ignored.
HOST_IP	Specifies the physical IP address of the current node.	Valid IP address in the format of string; must be in standard IPv4 or IPv6 format	Fixed configuration item, provided in the initial YAML file.

Elastic Agent Environment Variables

Note

Elastic Agent has reached its end of life, and related information will be removed in the version released on December 30, 2026.

The table below describes environment variables that can be configured when Elastic Agent is used. For other environment variables from the source code, see PyTorch Documentation.

Table 4 Elastic Agent environment variables

Environment Variable	Function	Value	Description
ELASTIC_LOG_PATH	Specifies the disk path for Elastic Agent running logs.	String	When configuring, differentiate the node name for this log. Reference example: ELASTIC_LOG_PATH=/job/code/alllogs/$MINDX_TASK_ID/elasticlogs/elastic-log$XDL_IP-$RANK Replace $XDL_IP with the actual node IP. Replace $RANK with the actual node RANK.
ELASTIC_PROCESS_RECOVER_ENABLE	Switch for Elastic Agent-side process-level rescheduling, process-level online recovery, and dying gasp checkpoint recovery.	String	1: Enable this feature. Other values: Disable this feature. When this feature is disabled, the related features on the MindIO side must also be disabled.
ENABLE_RESTART_FAULT_PROCESS	Switch for Elastic Agent to enable the process-level in-place recovery feature.	String	on: Enable this feature. Other values: Disable this feature.
RESTART_FAULT_PROCESS_TYPE	Type of faulty process restart that Elastic Agent notifies MindIO to perform.	String	worker: Do not exit the Pod, and only restart the faulty process. pod: Restart the Pod.
RANK_TABLE_FILE	RankTable file path.	String	Path to the hccl.json file.
PROCESS_RECOVER	Switch for process-level rescheduling or process-level online recovery.	String	on: Enable this feature. Other values: Disable this feature.

TaskD Environment Variables

The table below describes environment variables that can be configured when TaskD is used. For other environment variables from the source code, see PyTorch Documentation.

Table 5 TaskD environment variables

Environment Variable	Function	Value	Description
TASKD_LOG_PATH	Specifies the disk path for storing TaskD running logs.	String	If not specified, the default path is used: ./taskd_log/, which is the taskd_log directory under the current execution path. The following logs are generated based on different node configurations: manager.log: TaskD Manager log taskd.log: TaskD Python-side log agent-{RANK}.log: TaskD Agent log taskd-proxy-{RANK}-{TIMESTAMP}.log: TaskD Proxy log taskd-worker-{RANK}.log: TaskD Worker log {RANK} is the global rank number of the current training process, and {TIMESTAMP} is the timestamp.
TASKD_FILE_LOG_LEVEL	Specifies the log level to be recorded in the log file.	String	Value: DEBUG: Debug information INFO: General information (default level) WARNING: Warning information ERROR: Error information
TASKD_STD_LOG_LEVEL	Specifies the log level to be printed.	String	Value is: DEBUG: Debug information INFO: General information (default level) WARNING: Warning information ERROR: Error information
TASKD_LOG_STDOUT	Specifies whether logs need to be printed.	bool	Value is True or False. Defaults to True if not configured.
ENABLE_RESTART_FAULT_PROCESS	Switch for the TaskD component to enable the process-level in-place recovery feature.	String	Value: on: Enable this feature Other Values: Disable this feature
RESTART_FAULT_PROCESS_TYPE	Type of notification from TaskD to MindIO for restarting the faulty process.	String	Value: worker: Do not exit the pod, and only restart the faulty process pod: Restart the pod
TASKD_PROCESS_ENABLE	Switch for TaskD to enable process-level rescheduling, process-level online recovery, process-level in-place recovery, and elastic training features.	String	Value: on: Enable this feature off: Disable this feature
LOCAL_PROXY_ENABLE	Whether to enable the local proxy (required for security hardening).	String	Value: on: Enable this feature off: Disable this feature Default Value is "off". It must be set to "on" for communication security hardening scenarios.
HCCL_ASYNC_ERROR_HANDLING	Whether to enable the watchdog feature.	String	Value: 0: Disables fault detection and process exit features. 1: Enables fault detection and process exit features. 2: Enables only the fault detection feature. Default Value is 1.
TASKD_PROCESS_INTERVAL	Sets the processing interval for the TaskD Manager main process.	String	Value ranging from 100 to 1000, in milliseconds.
TASKD_REPORT_FAULT_TIMEOUT	Timeout for TaskD Agent to report service faults to TaskD Manager. If the fault persists after the timeout, TaskD Agent exits.	String	Value ranging from 300 to 600, in seconds.

NodeD Environment Variable

Table 6 NodeD environment variable

Environment Variable	Function	Value	Description
XDL_IP	Used to obtain the IP address of the host where the pod resides. Used by slow nodes to record and match slow node information.	A valid IP address in string format, either regular IPv4 or IPv6.	Write this environment variable in the YAML for deploying NodeD.