Common Operations

Installing Dependencies on UOS

MindCluster Ascend Deployer does not support installing the system dependencies required for Ascend software packages on UOS. Perform the following operations to install them manually.

In the yum repository configuration path /etc/yum.repos.d, create a cann_docker.repo file and write the following example content. Modify it according to your organization's security requirements and version needs.

# This needs to be added when installing drivers, firmware, and CANN software.
[docker-ce]
baseurl=https://mirrors.huaweicloud.com/docker-ce/linux/centos/7/aarch64/stable
gpgcheck=0
[extra]
baseurl=https://mirrors.huaweicloud.com/centos-altarch/7/extras/aarch64/
gpgcheck=0

Run the following command to install the dependencies required for the driver, firmware, CANN software, etc.

yum install blas-devel  gcc-c++ net-tools haveged sshpass dkms gcc kernel-devel-$(uname -r) make haveged bzip2 unzip tar docker-ce-18.09.0 docker-ce-cli-18.09.0 containerd.io-1.6.16 zlib-devel openssl-devel xz-devel bzip2-devel libffi-devel sqlite-devel bzip2-devel python3-libselinux expect hdf5-devel dos2unix

Manually install Kubernetes and configure the inventory_file, then proceed with the MindCluster installation.
- On UOS systems, the MindCluster cluster scheduling component only supports configuration with Kubernetes versions 1.19.16 to 1.28.X (cri-dockerd is required for Kubernetes 1.24 and later versions) and Docker 18.09.x and later versions. Otherwise, the Kubernetes configuration will be skipped, and the MindCluster cluster scheduling software package will be installed directly.
- To install the MindCluster cluster scheduling component on UOS 20-1020e and 20-1050u2e systems, first manually complete the Kubernetes installation and inventory_file configuration, and then use MindCluster Ascend Deployer to execute the MindCluster installation command.

Uninstalling MindCluster Ascend Deployer

MindCluster Ascend Deployer is an installation and deployment tool. It should be deleted immediately after system installation to free up drive space.

Clean up the ~/resources and ~/resources__{arch}_.tar files on the management server and remote machines.
```
bash install.sh --clean
```

Manually clean up and uninstall MindCluster Ascend Deployer.

If installed via a .zip package on Windows and Linux, delete the .zip file and the extracted directory during uninstallation.

If installed via pip on Linux, run the following command to uninstall.

pip3 uninstall ascend-deployer

After uninstalling the tool on Linux, you also need to delete the content shown in Table 1.

Table 1 Content to be deleted

Directory	Description
ascend-deployer	Directory of the offline deployment tool on the management server (MindCluster Ascend Deployer executor)
~/.ansible	Custom information collection configuration files on the management server and remote machines
~/build	Source code package decompression directory on the management server and remote machines

Setting the User Validity Period

To ensure user security, set the user validity period using the system command chage.

The command is:

chage [-m mindays] [-M maxdays] [-d lastday] [-I inactive] [-E expiredate] [-W warndays] user

Example: Modify the validity period of user test to 90 days.

chage -M 90 test

For related parameters, see Table 1.

Table 1 Setting the user validity period

Parameter	Parameter Description
-m	Minimum number of days between password changes. Setting this to `0` means the password can be changed at any time.
-M	Maximum number of days the password remains valid. Setting this to `-1` disables this password check. Setting it to `99999` means unlimited.
-d	Date of the last password change.
-I	Inactivity period. After the specified number of days past expiration, the password is set to an inactive state.
-E	Date on which the user account expires. After this date, the user will be unavailable.
-W	Number of days before password expiration that a warning message is sent.
-l	List current settings. Used by non-privileged users to determine when their password or account expires.

Note

Table 1 only lists common parameters. You can query detailed parameter descriptions using the chage --help command.
The date format is YYYY-MM-DD. For example, chage -E 2017-12-01 test means the password for user test expires on December 1, 2017.
user must be specified. Replace it with the specific user when executing the command. The default is the root user.

Installing and Rolling Back CANN Patches

Constraints

Patches only support upgrades for the corresponding baseline version or related patch versions.
For patches based on the same baseline version, ensure that the subsequently installed patch version is later than the previously installed patch version.
Only rollback of one patch version is supported.

Installation and Rollback Impact

Other maintenance operations are prohibited during the installation and rollback process.
Service interruption occurs during the software package installation and rollback process.
After the patch package is installed or rolled back, normal services will not be affected.

Procedure

The installation process is basically the same as Installing Ascend Software, with the main differences as follows:

CANN patches do not support online download using MindCluster Ascend Deployer. You need to obtain the required CANN patches and place them in the ascend-deployer/ascend_deployer/resources/patch directory (if the patch directory does not exist, create it). Note that before installation, delete the CANN software package and the run_from_cann_zip directory from the ascend-deployer/ascend_deployer/resources directory.
The commands for installing and rolling back CANN cold patches are as follows:
- Install the CANN cold patch (using the nnae package as an example):
```
bash install.sh --patch=nnae
```
- Roll back the CANN cold patch (using the nnae package as an example):
```
bash install.sh --patch-rollback=nnae
```

Configuring the Repository

MindCluster Ascend Deployer provides repository configuration files, which users can replace as needed.

Python repository configuration

Configure the Python repository in the ascend-deployer/ascend_deployer/downloader/config.ini file. The Huawei repository is used by default.

```shell
[pypi]
index_url=https://repo.huaweicloud.com/repository/pypi/simple
```

System repository configuration

The system repository configuration file is ascend-deployer/ascend_deployer/downloader/config/_{os}_{version}_{arch}_/source._xxx_.

Taking CentOS 7.6 AArch64 as an example, the content of the repository configuration file ascend-deployer/ascend_deployer/downloader/config/CentOS_7.6_aarch64/source.repo is as follows:

```shell
[base]
baseurl=https://mirrors.huaweicloud.com/centos-altarch/7/os/aarch64

[epel]
baseurl=https://mirrors.huaweicloud.com/epel/7/aarch64
```

This indicates that both the base and epel repositories are enabled. System components will be queried and downloaded from these two repositories. If a repository is deleted, component downloads may be incomplete. The Huawei repository is used by default and can be modified based on actual needs. Ensure that the repository complies with the security/vulnerability patching requirements of your organization.

If modifications are made, select a secure and reliable repository, and test whether the download and installation behavior is normal. Otherwise, it may cause incomplete component downloads or installation exceptions.

Note

When downloading CentOS-like system components, XML files within the system repository need to be parsed. It is recommended to install the defusedxml security component in the system Python3 to enhance security capabilities against potential XML vulnerability attacks.
If a resource does not exist when downloading system dependencies, you can modify the downloader/config/_{os}_{version}_{arch}_/pkg_info.json file to specify the resource acquisition link. The resource acquisition link must comply with the security requirements of your organization.

Limiting Process CPU Usage or Memory Usage (Linux)

If you want to limit the CPU usage or memory usage of MindCluster Ascend Deployer, refer to this section.

cgroups, short for control groups, is a mechanism provided by the Linux kernel that can limit the resources used by a single process or multiple processes, enabling fine-grained control over resources such as CPU and memory. The operation reference is as follows:

Limit the CPU used by MindCluster Ascend Deployer

Create a subdirectory.
```
cd /sys/fs/cgroup/cpu
mkdir ascend-deployer         # The directory name can be customized.
```
After the ascend-deployer directory is created, files such as cpu.cfs_period_us and cpu.cfs_quota_us are automatically generated in this directory.

Go to the ascend-deployer directory and set the values in the corresponding files. The CPU limitation mainly involves the following files.

Table 1 File description

File Name	Description
cpu.cfs_period_us	The period for counting CPU usage time, in microseconds (us). The value ranges from 1000 to 1000000, and the default value is `100000`.
cpu.cfs_quota_us	The allowed CPU time within the set period (referring to single-core time; for multi-core, the time needs to be accumulated during setting). The default value is `-1`, indicating no limit. For example, to limit to 30%, change the value to `30000`. For example, to limit the maximum usage to two CPU cores, change the value to `200000`.
tasks	A list of PIDs of the program processes to be limited. For child processes spawned by a PID after the limit takes effect, their PIDs will be automatically added to this file. It is recommended to add the PID of the shell running the program (execute the command `echo $$`) to `tasks` before starting the program.

Taking limiting the maximum CPU usage to 50% as an example, the operation is as follows:

Keep the default value in the cpu.cfs_period_us file, change the value of cpu.cfs_quota_us to 50000, execute echo $$ in the shell window where MindCluster Ascend Deployer is to be run to query its PID, and write it into the tasks file.

Limiting the running memory of MindCluster Ascend Deployer

Create a subdirectory.
```
cd /sys/fs/cgroup/memory
mkdir ascend-deployer      # The directory name can be customized.
```
After the ascend-deployer directory is created, files such as memory.limit_in_bytes and tasks are automatically generated in this directory.

Go to the ascend-deployer directory and set the values in the corresponding files. The following files are mainly involved in memory limiting.

Table 2 File description

File Name	Description
memory.limit_in_bytes	Limits the memory usage of a process. For example, to limit the maximum memory usage to 500 MB, the value should be 50010241024=524288000. Modify the value in the `memory.limit_in_bytes` file to the calculated value 524288000.
tasks	A list of PIDs of the processes to be limited. For child processes spawned by a PID after the limit takes effect, the PIDs of the child processes are automatically added to this file. It is recommended that you add the PID of the shell running the program (run the command `echo $$`) to `tasks` before starting the program.

For example, to limit the maximum memory usage to 500 MB, modify the value in the memory.limit_in_bytes file to the calculated value 524288000. In the shell window where MindCluster Ascend Deployer is to be run, run echo $$ to query its PID and write the PID to the tasks file.

Configuring SSH Timeout

Some OSs set a timeout interval by default. You can modify the TMOUT environment variable and the related information in the sshd_config file to avoid frequent timeout prompts.

Modify the TMOUT environment variable. The value 0 indicates no timeout interval. You can set it to a positive integer, indicating the maximum idle time of a session (in seconds). For example, setting TMOUT to 600 means that if you have no activity within 10 minutes, your login status will be automatically terminated.
```
export TMOUT=600
```
Modify the ClientAliveInterval and ClientAliveCountMax fields in the sshd_config file according to your organization's security requirements.

Performing a Pre-installation Check

Scenario Description

The dependencies required by Ascend software packages are complex. To improve installation efficiency, MindCluster Ascend Deployer provides a pre-installation check function. Before executing the installation command, users can run the following command to test whether installation is supported based on the installation scenario or software package, and then perform the installation operation after the test passes.

In the following example, --stdout_callback=ansible_log is an optional parameter used to enable the on-screen display plugin.

Pre-installation Check for Specified Scenarios (Root User Only)

MindCluster Ascend Deployer provides several basic installation scenarios. For details about <scene_name>, see Supported Installation and Upgrade Scenarios. You can run the following command to perform a pre-installation check.

bash install.sh --install-scene=<scene_name> --check --stdout_callback=ansible_log

The command example is as follows:

bash install.sh --install-scene=dl --check             #Test whether cluster scheduling installation is supported

Specifying Pre-installation Checks

For the optional range of <package_name>, refer to Table 1, or run bash install.sh --help to view it.

bash install.sh --install=<package_name> --check --stdout_callback=ansible_log

The following is a command example:

bash install.sh --install=toolkit --check --stdout_callback=ansible_log          #Test whether the installation of the toolkit is supported.

Outputting Check Result

After the check is complete, MindCluster Ascend Deployer outputs the check result file check_res_output.json in the ~/.ascend_deployer/deploy_info directory.

Installing Ascend Software (Using Self‑prepared Packages)

If you have downloaded the commercial or community version of Ascend software, refer to this section for installation.

Prerequisites

MindCluster Ascend Deployer has been downloaded.
The software packages have been prepared.
The prepared software packages must be within the installation scope supported by MindCluster Ascend Deployer, as shown in Table 1.
You need to prepare a login account for the enterprise support website in advance, and apply for and confirm the download permissions for the corresponding software packages (driver and firmware packages, CANN software packages, and MindCluster performance testing software packages). Applications can be redirected via the URL links in the ascend-deployer/ascend_deployer/downloader/software/CANN_*.json files.

Procedure

Download Docker software and system dependencies. Double-click start_download_ui.bat in the ascend-deployer-{version}/ascend_deployer directory. In the interface, select the OS of the device to be installed. Do not select the software on the right side of the interface (PKG_LIST).
(Optional) Select Enable Proxy and fill in the proxy information.

On Windows 10 and later operating systems, users can use a configured proxy server for software downloads. This feature is not supported on other operating systems.
Click Download.

The following output indicates that the download is complete, as shown in Figure 2.

Figure 2 Download complete

Place the prepared software packages in the following paths.

Place software packages such as NPU drivers and firmware, toolkit, kernels, and MindCluster performance testing in the ascend-deployer-{version}/ascend_deployer/resources directory.

The contents of the resources folder are as follows (using CentOS 7.6 as an example):
```
 resources
 ├── CANN_8.2.RC1
 ├── FaultDiag_7.1.RC1
 ├── MindIE-image
 ├── mindxdl
 ├── nexus
 ├── npu
 ├── CentOS_7.6_aarch64
 ├── pylibs
 ├── sources
 └── version.json
```

(Optional) Replace the MindCluster component packages in the ascend_deployer/resources/mindxdl/dlPackage/{arch} folder with the downloaded MindCluster component packages.

The contents of the ascend_deployer/resources/mindxdl/dlPackage/{arch} folder are as follows.

 ascend_deployer/resources/mindxdl/dlPackage/{arch}
 ├─ Ascend-docker-runtime_{version}_linux-{arch}.run
 ├─ Ascend-mindxdl-ascend-operator_{version}_linux-{arch}.zip
 ├─ Ascend-mindxdl-infer-operator_{version}_linux-{arch}.zip
 ├─ Ascend-mindxdl-device-plugin_{version}_linux-{arch}.zip
 ├─ Ascend-mindxdl-noded_{version}_linux-{arch}.zip
 ├─ Ascend-mindxdl-npu-exporter_{version}_linux-{arch}.zip
 ├─ Ascend-mindxdl-resilience-controller_{version}_linux-{arch}.zip
 └─ Ascend-mindxdl-volcano_{version}_linux-{arch}.zip
 └─ Ascend-mindxdl-clusterd_{version}_linux-{arch}.zip

Note

{version} indicates the software version number.
{arch} indicates the CPU architecture.

Compress the entire ascend-deployer-{version} directory into a .zip package and upload it to any directory on the executor (the server used to deploy MindCluster Ascend Deployer for batch installation).
Run the unzip ascend-deployer-{version}.zip command to decompress and obtain the ascend-deployer-{version} directory.

Manually Creating a Running User

Ascend software requires the creation of a special running user. If you need to create one manually, refer to the following operations.

The command reference is as follows (using the creation of the HwHiAiUser as an example):

If you want to specify another user as the running user, modify the configuration file inventory_file during the subsequent installation operations.

Create the running user HwHiAiUser.

groupadd HwHiAiUser
useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash

Set the password for the HwHiAiUser user.
```
passwd HwHiAiUser
```

Note The password must meet the password complexity requirements (see Password Complexity Requirements). The password validity period is 90 days. You can modify the number of days for the validity period in the /etc/login.defs file, or use the chage command to set the user validity period. For details, see Setting the User Validity Period.

Configuring DNS

Run the cat /etc/resolv.conf command on all devices to be installed to check whether the OS has been configured with DNS. It is recommended that the DNS settings be consistent across all devices to be installed.

If DNS configuration information (such as nameserver information) is displayed, DNS has been configured.

If no information is displayed, DNS has not been configured. Perform the following steps to configure it.

Query the name of the network interface configured with the service IP address.
```
ip a
```
Configure DNS for the corresponding network interface. It is recommended that the DNS settings be consistent across all network interfaces.

Example: The network interface name queried from Step 1 is enp125s0f0.
```
nmcli connection modify enp125s0f0 +ipv4.dns 10.10.10.254
nmcli connection up enp125s0f0
```
Note
- 10.10.10.254 is the DNS address and is only an example.
- enp125s0f0 is the network interface name and is only an example.
- Using the nmcli command to configure DNS is only an example. You must complete the configuration according to the DNS configuration scheme provided by the OS image vendor.
Check whether the DNS is configured successfully.
```
cat /etc/resolv.conf
```
The following information is displayed, indicating that the DNS is configured successfully.
```
# Generated by NetworkManager
nameserver 192.0.2.254
```

Configuring other_build_image

When batch installing and upgrading cluster scheduling components, MindCluster Ascend Deployer supports batch installation and upgrade of MindCluster cluster scheduling on heterogeneous nodes (worker nodes with an architecture different from the executor architecture). Select a heterogeneous node as the build server for heterogeneous images, and complete the configuration of other_build_image (heterogeneous server information) in the inventory_file file on the build server.

Heterogeneous Scenario Description

The heterogeneous scenarios supported by MindCluster Ascend Deployer are as follows.

Executor Architecture	Server Architecture of the Master Node	Server Architecture of the Worker Node
x86	x86	ARM

Configuring other_build_image

Table 1 other_build_image parameter description

Parameter	Mandatory/Optional	Description
IP	Mandatory	Configure the IP address of the server where the heterogeneous node resides.
ansible_ssh_user	Mandatory	The account for the heterogeneous node to log in to the remote server via SSH. It must be the root account.
ansible_ssh_pass	Optional	The password for the heterogeneous node's SSH login account to the remote server. If SSH key authentication is configured and the root user can log in, this is not required.
set_hostname	Optional, but required for multiple master or worker nodes; optional for a single node	Sets the node name of the heterogeneous node in the Kubernetes cluster. It is recommended to use the format master-1 or worker-1 and fill in sequentially. If a Kubernetes cluster already exists, this name must be the heterogeneous node's name in Kubernetes, must be configured in lowercase, and cannot be filled in arbitrarily.

Installing and Deploying Ultra-Large-Scale Clusters

MindCluster Ascend Deployer supports the installation and deployment of ultra-large-scale clusters with over 100,000 cards within a single day through multi-instance deployment.

To improve deployment efficiency, a large-scale cluster is divided into multiple sub-clusters. One node is selected from each sub-cluster as an instance node, which serves as a distributed deployment node. The master node then collects and summarizes the deployment information from all instance nodes.

Note

Multi-instance deployment does not support simultaneous operation on two operating systems.

When using MindCluster Ascend Deployer by downloading the zip package and extracting it, you need to enter the ascend_deployer directory and run the bash large_scale_install.sh --install command to install and deploy an ultra-large-scale cluster.

bash large_scale_install.sh --install=<package_name_1>,<package_name_2>

For a command example, see Table 1.

Perform the installation in the following order: sys_pkg > python > npu > CANN, MindCluster (performance testing, fault diagnosis, cluster scheduling). The CANN package version in the resources directory must be compatible with the NPU during installation.

Table 1 Installation command examples

Installation Type	Installation Command
System environment initialization (installing sys_pkg)	bash large_scale_install.sh --install=sys_pkg #When executing the --install command, do not install sys_pkg repeatedly.
System environment initialization (installing python)	bash large_scale_install.sh --install=python
NPU firmware and driver (execute either of the commands on the right)	bash large_scale_install.sh --install=npu bash large_scale_install.sh --install=driver,firmware
CANN software (training & inference & development and debugging scenarios)	bash large_scale_install.sh --install=kernels,toolkit
CANN software (edge inference scenario)	bash large_scale_install.sh --install=nnrt,kernels
CANN software (training & inference scenarios)	bash large_scale_install.sh --install=nnae,kernels
MindCluster cluster scheduling	bash large_scale_install.sh --install=ascend-device-plugin,ascend-docker-runtime,noded,npu-exporter,volcano,ascend-operator,infer-operator,clusterd,resilience-controller
MindCluster performance testing	bash large_scale_install.sh --install=toolbox
MindCluster fault diagnosis	bash large_scale_install.sh --install=fault-diag

(Optional) When the installation involves CANN and ToolBox, users need to sign the Huawei Enterprise Business End User License Agreement (EULA) before entering the installation process. Follow the on-screen prompts to enter y or Y to confirm the agreement, or enter any other character to reject it. The installation will start automatically after the agreement is accepted.

If the current language environment does not meet the requirements, you can run the following command to configure the system's default language environment.
- Configure to Chinese
```
export LANG=zh_CN.UTF-8
```
- Configure to English
```
export LANG=en_US.UTF-8
```

Viewing Installation Reports, Verification Reports, and Status Information

If the installation fails, a report directory will be generated under the ~/.ascend_deployer/large_scale_deploy/ path, containing the installation report files large_scale_deploy.json and host_deploy_report.csv. The report files record result information such as server IP addresses and statuses, organized by server.

An installation progress information file deployer_progress_output.json will be generated under /root/.ascend_deployer/large_scale_deploy/remote_host_data/\{IP}/ for viewing the installation process and status information.

Run the following command. A verification report, test_report.csv, will be generated in the report directory.

bash large_scale_install.sh --test=all

The report file records version information such as the server IP address and Ascend software, with the server as the dimension.

Configuring Large-Scale Deployment Files

Configure the IP address and username of the device to be installed on the MindCluster Ascend Deployer executor.

Go to the ascend-deployer/ascend_deployer directory, edit the large_scale_inventory.ini file, and after adding the content, execute :wq to save and exit.

According to Table 2, complete the variable configuration for master, worker, deploy_node (optional), and npu_node (optional).

Note

The Kubernetes version must be 1.28 and later versions.
When applying the MindCluster cluster scheduling configuration to the Kubernetes cluster, you need to pay attention to the different NPU hardware forms in the cluster and provide the characteristic server information for different hardware. Configure the characteristic server information for different hardware by setting npu_node, and fill in the information of any one characteristic server for each type of hardware. If npu_node is not configured, the default is the NPU hardware type of the first node configured under worker in large_scale_inventory.ini. npu_node must be in the worker group, and you can fill in only the IP. The parameters of the worker group will overwrite the npu_node parameters according to the IP correspondence.

Table 2 Parameter description

Parameter	Mandatory/Optional	Description
IP	Mandatory	IP address of the server.
ansible_ssh_user	Mandatory	Account for SSH login to the remote server, which must be the root account.
ansible_ssh_pass	Optional	Password for the SSH login account to the remote server. If SSH key authentication is configured and the root user can log in, this is not required. In large-scale deployment scenarios, password-free access must be configured among all nodes in the cluster.
ansible_ssh_port	Optional	Port for the SSH connection. Not required if the default port 22 is used. Required if a non-default port is used.
set_hostname	Optional. Mandatory for multiple master or multiple worker nodes. Optional for a single node.	Sets the node name in the Kubernetes cluster. It is recommended to use the format master-1 or worker-1 and fill in sequentially. If a Kubernetes cluster already exists, this name must be the node's name in Kubernetes, must be configured in lowercase, and cannot be filled in arbitrarily.
npu_num	Optional	Sets the number of NPUs.
index	Optional	The sequence number of the server in the IP range, of str type. For example, for `1.1.1.1-1.1.1.3 set_hostname="master-{index}"`, the corresponding `set_hostname` for `1.1.1.1` is `master-1`.
step_len	Optional	Step length for IP skipping. In actual deployment, IP range skipping may occur. For example, for `1.1.1.1, 1.1.1.3, 1.1.1.5`, the step length is 2. Example 1: `1.1.1.1-1.1.1.5 step_len=2`. The output is `1.1.1.1, 1.1.1.3, 1.1.1.5`. Example 2: `1.1.1.1-1.1.1.6 step_len=2`. The output is `1.1.1.1, 1.1.1.3, 1.1.1.5, 1.1.1.6`. That is, the last IP is retained even if it is not within the step length.

Note

The host configured as the master node is the default control node for Kubernetes.
The number of masters must be an odd number.
Parameter parsing is supported within {} in the batch input configuration. Basic mathematical operations and type casting to str and int are supported. The final output after {} parsing is a string type.

For example, for 1.1.1.1-1.1.1.3 set_hostname="master-{ str(int(index)+int('20')) + 'x'}", the parsed host information is 1.1.1.1 set_hostname="master-21x" 1.1.1.2 set_hostname="master-22x"…

Complete the large-scale deployment parameter configuration under the [large_scale] configuration field.

Table 3 Parameter description

Parameter	Mandatory/Optional	Description
SUB_GROUP_MAX_SIZE	Optional	Maximum size of a sub-cluster. When dividing sub-clusters, ensures that the size of each sub-cluster is less than or equal to this value. Int type. The default value is `200`.

Example:

 [master]
[worker]
xx.xxx.xx.x1-xx.xxx.xx.x9 ansible_ssh_user="root" ansible_ssh_pass="xxxxxxx"  step_len=3 set_hostname="master-{ip}-{int(index)+1}-y"
[deploy_node]
10.1.1.1
[npu_node]   #npu_node is used to set the characteristic server information for different hardware. For each hardware type, fill in the information of any one characteristic server.
[large_scale]
SUB_GROUP_MAX_SIZE=5
[all:vars]

deploy_node supports the following configuration methods
1. Manual configuration: Manually fill in parameters. This method has the highest priority.
2. Automatic configuration: If no host in the [deploy_node] host group is manually specified, the automatic selection mode is entered.
  
  The selection method is as follows:

When the maximum value of SUB_GROUP_MAX_SIZE in [large_scale] in large_scale_inventory.ini is 200, the tool automatically sorts IPs in order, divides them into sub-clusters with a maximum of 200, and uses the first server of each cluster as the instance node to install and deploy Ascend software.

Note

On Atlas A2 Training Series products, IP supports configuration of both IPv4 and IPv6 address types. The IP address type used by the user to connect to the executor via an SSH client such as PuTTY must be consistent with the IP address type configured in the large_scale_inventory.ini file, both being IPv4 or both being IPv6. Other devices only support configuration with IPv4 addresses.

Configuring Deployment Parameters for Server Installation or Upgrade via an Excel Spreadsheet

For supported hardware products, OS lists, and installation scenarios when configuring installation or upgrade parameters, refer to Supported Products and OSs.

When batch installing or upgrading NPU firmware and drivers, CANN software packages, AI frameworks, MindCluster components (performance testing, fault diagnosis, cluster scheduling), MindIE images, and configuring HCCN parameters, you can import the parameters required for the inventory file by filling in an Excel spreadsheet. This configuration method is only supported on Windows.

Downloading the Excel

The table is named inventory_template.xls. It is downloaded together with the MindCluster Ascend Deployer installation software package and stored in the ascend_deployer directory.

Using the Excel

Read the table filling instructions carefully and fill in the table completely as required.
Save the completed table in CSV UTF-8 format.
Place the saved CSV file into the ascend_deployer directory.
Run the trans_excel_to_inventory.bat script (this script is also stored in the ascend_deployer directory) to extract the CSV content and save it to inventory_file.

Using Docker or Kubernetes to Start a Container for NPU Driver Installation

When the server has docker, kernel-headers, and kernel-dev, you can install the NPU driver by pulling up the ascend-npu-driver-installer container. For details, see How to Install NPU Drivers Using Containers.
When you have a Kubernetes cluster, you can use Kubernetes to pull up pods to install the NPU driver on all nodes. For details, see Installing NPU Drivers Through a K8s Cluster.

It is incompatible with the NPU driver installation by MindCluster Ascend Deployer. That is, after installing the NPU driver using containerization, you cannot use MindCluster Ascend Deployer for upgrades or checks, and vice versa.