Star181
187
代码
代码
Issues136
Pull Requests95
流水线
项目讨论
Wiki
分析
Star181
187
ascend-robotascend-robot【docs】合入集群调度英文资料
dfa1b8a5创建于 7月3日历史提交

Common Operations

Scheduling Configuration

Volcano supports native K8s scheduling and can use nodeAffinity for scheduling. The following example uses mandatory node affinity for scheduling. For more information about the nodeAffinity field, see the official Kubernetes official.

  • In the Volcano Job YAML, add the following fields in bold.

      apiVersion: batch.volcano.sh/v1alpha1
  kind: Job
  metadata:
    name: mindx-test
    labels:
  ...
  spec:
  ...
    maxRetry: 3
    queue: default
    tasks:
    - name: "default-test"
      replicas: 1
      template:
        metadata:
          labels:
  ...
        spec:
          affinity:      # Add the following fields in bold
            nodeAffinity:                             # Node affinity configuration
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:                    # Node selector list
                  - matchExpressions:
                      - key: aaa               # Match nodes with label key "aaa" and value "yyy"
                        operator: In
                        values:
                          - yyy
               podAntiAffinity:
                 requiredDuringSchedulingIgnoredDuringExecution:
  ...
            nodeSelector:
              example-key: example-value    # Example value.Configure nodeSelector based on scheduling intent.
  ...

  • In the Ascend Job YAML, add the following fields in bold.

      apiVersion: mindxdl.gitee.com/v1
  kind: AscendJob
  metadata:
    name: test-2
  ...
  spec:
    schedulerName: volcano
    runPolicy:
      schedulingPolicy:
        minAvailable: 2
        queue: default
    successPolicy: AllWorkers
    replicaSpecs:
      Master:
        replicas: 1
        restartPolicy: Never
        template:
          metadata:
            labels:
  ...
          spec:
            affinity:   #  New field
              nodeAffinity:                           # Node affinity configuration
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:                  # Node selector list
                    - matchExpressions:
                      - key: aaa            # Match nodes with label key "aaa" and value "yyy"
                        operator: In
                        values:
                          - yyy
            nodeSelector:
              example-key: example-value    # Optional value. Configure nodeSelector based on actual requirements.
  ...

    Note

    You can query node labels by running the kubectl get node --show-labels command. In the LABELS field, the value before the equal sign is the label key, and the value after the equal sign is the label value, for example, aaa=yyy.

Installing NFS

Ubuntu OS

Network File System (NFS) allows computers in a network to share resources. In cluster scheduling scenarios, an NFS environment is required to ensure the normal operation of training or inference jobs. NFS can be installed on the server side or the client side, and you can choose as needed.

Installing on the Server

  1. Log in to the storage node using an administrator account and run the following command to install the NFS server.

    apt install -y nfs-kernel-server

  2. Fix the NFS-related ports based on the actual situation and configure the firewall for the relevant ports.

  3. Create a shared directory (such as /data/atlas_dls) and modify the directory permissions.

    mkdir -p /data/atlas_dls
chmod 750 /data/atlas_dls/

  4. Append the following content to the end of the /etc/exports file, configure the allowed IP addresses as needed, and strengthen the relevant permission settings.

    /data/atlas_dls Service IP address (configure the necessary permission)

  5. Start rpcbind.

    systemctl restart rpcbind.service
systemctl enable rpcbind.service

  6. Run the following command to check whether rpcbind has been started.

    systemctl status rpcbind.service

    If information similar to the following appears, the service is normal.

    ● rpcbind.service - RPC bind portmap service
   Loaded: loaded (/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2024-01-08 16:39:03 CST; 6 days ago
     Docs: man:rpcbind(8)
 Main PID: 2952 (rpcbind)
    Tasks: 1 (limit: 29491)
   CGroup: /system.slice/rpcbind.service
           └─2952 /sbin/rpcbind -f -w


Jan 08 16:39:03 ubuntu-211 systemd[1]: Starting RPC bind portmap service...
Jan 08 16:39:03 ubuntu-211 systemd[1]: Started RPC bind portmap service.

  7. After rpcbind starts, start the NFS service.

    systemctl restart nfs-server.service
systemctl enable nfs-server.service

  8. Check whether the NFS service has started.

    systemctl status nfs-server.service

    If information similar to the following appears, the service is normal. If the NFS service fails to start, see the df -h execution failure, causing NFS startup failure section for troubleshooting.

    ● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2024-01-08 16:39:03 CST; 6 days ago
 Main PID: 3220 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 29491)
   CGroup: /system.slice/nfs-server.service


Jan 08 16:39:03 ubuntu-211 systemd[1]: Starting NFS server and services...
Jan 08 16:39:03 ubuntu-211 exportfs[3181]: exportfs: /etc/exports [1]: Neither 'subtree_check' or 'no_subtree_check' specified for export "*:/data/atlas_dls".
Jan 08 16:39:03 ubuntu-211 exportfs[3181]:   Assuming default behaviour ('no_subtree_check').
Jan 08 16:39:03 ubuntu-211 exportfs[3181]:   NOTE: this default has changed since nfs-utils version 1.0.x
Jan 08 16:39:03 ubuntu-211 systemd[1]: Started NFS server and services.

  9. View the mount permissions of the shared directory (such as /data/atlas_dls).

    cat /var/lib/nfs/etab

    If information similar to the following appears, the service is normal.

    /data/atlas_dls *(rw,...Configured permission)

Installing on the Client

Log in to other servers using an administrator account and run the following command to install the NFS client.

apt install -y nfs-common

CentOS OS

NFS is a network file system that allows computers on a network to share resources. In cluster scheduling scenarios, an NFS environment is required to ensure the normal running of training or inference tasks. NFS can be installed on the server or the client, and you can choose based on your needs.

Install on the server

  1. Log in to the storage node using an administrator account and run the following command to install the NFS server.

    yum install nfs-utils -y

  2. Fix the NFS-related ports and configure the firewall for these ports based on your actual situation.

  3. Create a shared directory (such as /data/atlas_dls) and modify the directory permissions.

    mkdir -p /data/atlas_dls
chmod 750 /data/atlas_dls/

  4. Run the vi /etc/exports command, append the following content to the end of the file, configure the allowed IP addresses as needed, and strengthen the relevant permission settings.

    /data/atlas_dls Service IP address (configure necessary permission)

  5. Start rpcbind.

    systemctl restart rpcbind.service
systemctl enable rpcbind.service

  6. Check whether rpcbind has started.

    systemctl status rpcbind.service

    If information similar to the following appears, the service is normal.

    ● rpcbind.service - RPC bind service
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2024-01-15 15:54:44 CST; 28s ago
 Main PID: 63008 (rpcbind)
   CGroup: /system.slice/rpcbind.service
           └─63008 /sbin/rpcbind -w


Jan 15 15:54:44 centos39 systemd[1]: Starting RPC bind service...
Jan 15 15:54:44 centos39 systemd[1]: Started RPC bind service.

  7. After rpcbind starts, start the NFS service.

    systemctl restart nfs-server.service
systemctl enable nfs-server.service

  8. Check whether the NFS service has started.

    systemctl status nfs-server.service

    If information similar to the following appears, the service is normal. If the NFS service fails to start, see the df -h execution failure, causing NFS startup failure section for troubleshooting.

    ● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Fri 2024-01-15 15:56:15 CST; 8s ago
 Main PID: 67145 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service


Jan 15 15:56:15 centos39 systemd[1]: Starting NFS server and services...
Jan 15 15:56:15 centos39 systemd[1]: Started NFS server and services.

  9. View the mount permissions of the shared directory (e.g., /data/atlas_dls).

    cat /var/lib/nfs/etab

    If the following response appears, the service is normal.

    /data/atlas_dls *(rw,...Configured permission)

Installing on the Client

  1. Log in to other servers using an administrator account and run the following command to install the NFS client.

    yum install nfs-utils -y

  2. Start rpcbind.

    systemctl restart rpcbind.service
systemctl enable rpcbind.service

  3. Check whether rpcbind is started.

    systemctl status rpcbind.service

    If information similar to the following appears, the service is normal.

    ● rpcbind.service - RPC Bind
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2024-03-14 04:59:22 EDT; 8s ago
     Docs: man:rpcbind(8)
 Main PID: 1681425 (rpcbind)
    Tasks: 1 (limit: 3355442)
   Memory: 956.0K
   CGroup: /system.slice/rpcbind.service
           └─1681425 /usr/bin/rpcbind -w -f
Mar 14 04:59:22 localhost.localdomain systemd[1]: Starting RPC Bind...
Mar 14 04:59:22 localhost.localdomain systemd[1]: Started RPC Bind.

  4. After rpcbind is started, start the NFS service.

    systemctl restart nfs-server.service
systemctl enable nfs-server.service

  5. Check whether the NFS service is started.

    systemctl status nfs-server.service

If information similar to the following appears, the service is normal.

```ColdFusion
● nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled)
  Drop-In: /run/systemd/generator/nfs-server.service.d
           └─order-with-mounts.conf
   Active: active (exited) since Thu 2024-03-14 04:59:40 EDT; 8s ago
 Main PID: 1681567 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 3355442)
   Memory: 0B
   CGroup: /system.slice/nfs-server.service
Mar 14 04:59:39 localhost.localdomain systemd[1]: Starting NFS server and services...
Mar 14 04:59:39 localhost.localdomain exportfs[1681536]: exportfs: Failed to stat /data/atlas_dls: No such file or directory
Mar 14 04:59:40 localhost.localdomain systemd[1]: Started NFS server and services.
```

  1. (Optional) NFS requires the mount and umount commands. Generally, the system has the mount command built-in. If the current client does not have this command, run the following command.

    yum install -y  util-linux

Querying Reported Fault Information

Volcano

Volcano collects internal chip faults, parameter plane network faults, and node fault information, and places them as external information in the K8s ConfigMap for external query and use.

The query command is kubectl describe cm -n volcano-system vcjob-fault-npu-cm. The command response example is as follows. For key parameterdescriptions, see Table 2 vcjob-fault-npu-cm field description.

Name:         vcjob-fault-npu-cm
Namespace:    volcano-system
Labels:       <none>
Annotations:  <none>

Data
====
fault-node:
----
[{"FaultDeviceList":[{"fault_type":"CardNetworkUnhealthy","npu_name":"Ascend910-0","fault_level":"PreSeparateNPU","fault_handling":"PreSeparateNPU","large_model_fault_level":"PreSeparateNPU","fault_code":"81078603"},{"fault_type":"CardUnhealthy","npu_name":"Ascend910-4","fault_level":"SeparateNPU","fault_handling":"SeparateNPU","large_model_fault_level":"SeparateNPU","fault_code":"A8028801,A4028801,80E18402,80E18401"}],"NodeName":"node133","UnhealthyNPU":["Ascend910-4"],"NetworkUnhealthyNPU":["Ascend910-0"],"NodeDEnable":true,"NodeHealthState":"CardUnhealthy","UpdateTime":1744182212}]
remain-retry-times:
----


BinaryData
====

Events:  <none>

Ascend Device Plugin

Fault Information

Ascend Device Plugin collects internal chip faults, parameter plane network faults, and node faults, and places them as external information in K8s ConfigMaps. One ConfigMap stores the information of one node for external query and use.

Query command: kubectl describe cm -n kube-system mindx-dl-deviceinfo-$*_\{node\_name\}_

Taking Atlas A3 Training Series Products as an example, the response example is as follows. The response parameters may vary for different devices, and the actual output shall prevail. For key parameter descriptions, see Table 1 DeviceInfoCfg.

{"DeviceInfo":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-0,Ascend910-1,Ascend910-2,Ascend910-3,Ascend910-5,Ascend910-6,Ascend910-7","huawei.com/Ascend910-Fault":"[{\"fault_type\":\"CardNetworkUnhealthy\",\"npu_name\":\"Ascend910-0\",\"large_model_fault_level\":\"PreSeparateNPU\",\"fault_level\":\"PreSeparateNPU\",\"fault_handling\":\"PreSeparateNPU\",\"fault_code\":\"81078603\",\"fault_time_and_level_map\":{\"81078603\":{\"fault_time\":1744168468259,\"fault_level\":\"PreSeparateNPU\"}}},{\"fault_type\":\"CardUnhealthy\",\"npu_name\":\"Ascend910-4\",\"large_model_fault_level\":\"SeparateNPU\",\"fault_level\":\"SeparateNPU\",\"fault_handling\":\"SeparateNPU\",\"fault_code\":\"A8028801,A4028801,80E18402,80E18401\",\"fault_time_and_level_map\":{\"80E18401\":{\"fault_time\":1744167455784,\"fault_level\":\"NotHandleFault\"},\"80E18402\":{\"fault_time\":1744167455784,\"fault_level\":\"SeparateNPU\"},\"A4028801\":{\"fault_time\":1744167455784,\"fault_level\":\"NotHandleFault\"},\"A8028801\":{\"fault_time\":1744167455784,\"fault_level\":\"SeparateNPU\"}}}]","huawei.com/Ascend910-NetworkUnhealthy":"Ascend910-0","huawei.com/Ascend910-Recovering":"","huawei.com/Ascend910-Unhealthy":"Ascend910-4"},"UpdateTime":1744182144},"SuperPodID":-2,"ServerIndex":-2,"CheckCode":"a550811fdfafb5717555526816af2ca4ac6c3e102f5907574048578e0c8fcc73"}

Fault Event Information

Fault events collected by Ascend Device Plugin can be reported through K8s event events. The query command is kubectl get events -n kube-system. Taking Atlas Training Series Products as an example, the response example is as follows. For parameter descriptions, see Table 1.

NAMESPACE     LAST SEEN   TYPE      REASON     OBJECT                                         MESSAGE
kube-system   8s          Warning   Occur      pod/ascend-device-plugin-daemonset-910-dlpmv   device fault, nodeName:k8smaster, assertion:Occur, cardID:2, deviceID:0, faultCodes:8C084E00, faultLevelName:RestartBusiness, alarmRaisedTime:2023-11-21 05:36:53

Table 1 Parameter description

Name Description
NAMESPACE Namespace name, with the value kube-system.
LAST SEEN Time when the event occurred.
TYPE

Event type, with values of "Normal" and "Warning".
REASON

Reason for the event. The values are described as follows:

  • Occur: fault occurrence
  • Recovery: Fault recovery
  • Notice: Notification
OBJECT

Event object, with the value specification of pod/Ascend Device Plugin Pod name, such as pod/ascend-device-plugin-daemonset-910-dlpmv.
MESSAGE

Description of the event information content. The fields of the event content are described as follows:

  • nodeName: node name
  • assertion: information type
    • Occur: fault occurrence
    • Recovery: Fault recovery
    • Notice: Notification
  • cardID: NPU management unit ID (NPU device ID)
  • deviceID: Device number
  • faultCodes: Fault code, such as 8C084E00
  • faultLevelName: Fault level name
    • NotHandleFault: no handling required
    • RestartRequest: affect services execution; need to re-execute service requests
    • RestartBusiness: affect services execution; need to restart services
    • FreeRestartNPU: affect services execution; need to reset the chip when it is idle
    • RestartNPU: directly reset the chip and re-execute services
    • SeparateNPU: isolate chip
    • PreSeparateNPU: does not affect services temporarily, and no more jobs will be scheduled to this chip subsequently.
    • SubHealthFault: dandled according to the subHealthyStrategy parameter configured in the job YAML
  • alarmRaisedTime: fault occurrence time

ClusterD

ClusterD collects internal node faults, chip faults, and UnifiedBus device faults, and places them as external information in the K8s ConfigMap for external query and use.

Node Fault

Query command: kubectl describe cm -n mindx-dl cluster-info-node-cm

Taking the Atlas A3 Training Series products as an example, the response example is as follows. The response parameters may vary for different devices, and the actual output shall prevail. For key parameter descriptions, see Table 1 cluster-info-node-cm.

{"mindx-dl-nodeinfo-kwok-node-0":{"FaultDevList":[],"NodeStatus":"Healthy","CmName":"mindx-dl-nodeinfo-kwok-node-0"},"mindx-dl-deviceinfo-kwok-node-1001":{"FaultDevList":[],"NodeStatus":"Healthy","CmName":"mindx-dl-nodeinfo-kwok-node-1001"}}

Chip Fault

Query command: kubectl describe cm -n mindx-dl cluster-info-device-${m}

m is an integer incrementing from 0. For every additional 1000 nodes in ae cluster, a new ConfigMap file cluster-info-device-$\{m\} is added.

Taking the Atlas A3 Training Series Products as an example, the response example is as follows. The response parameters may vary for different devices, and the actual output shall prevail. For key parameter descriptions, see Table 2 cluster-info-device-${m}.

{"mindx-dl-deviceinfo-kwok-node-0":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-0,Ascend910-1,Ascend910-2,Ascend910-3,Ascend910-4,Ascend910-5,Ascend910-6,Ascend910-7","huawei.com/Ascend910-NetworkUnhealthy":"","huawei.com/Ascend910-Unhealthy":""},"UpdateTime":1693899390,"CmName":"mindx-dl-deviceinfo-kwok-node-0","SuperPodID":0,"ServerIndex":0},"mindx-dl-deviceinfo-kwok-node-1001":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-0,Ascend910-1,Ascend910-2,Ascend910-3,Ascend910-4,Ascend910-5,Ascend910-6,Ascend910-7","huawei.com/Ascend910-NetworkUnhealthy":"","huawei.com/Ascend910-Unhealthy":""},"UpdateTime":1693899390,"CmName":"mindx-dl-deviceinfo-kwok-node-1001","SuperPodID":0,"ServerIndex":0}}

UnifiedBus Device Fault

Query command: kubectl describe cm -n mindx-dl cluster-info-switch-${m}

m is an integer incrementing from 0. For every additional 2000 nodes in a cluster, a new ConfigMap file cluster-info-switch-$\{m\} is created.

Taking Atlas A3 Training Series Products as an example, the response example is as follows. The response parameters may vary for different devices, and the actual output shall prevail. For key parameter descriptions, see Table 1.

{"FaultCode":[000001c1],"FaultLevel":"NotHandle","UpdateTime":1722845555,"NodeStatus":"Healthy"}

Table 1 Lingqu bus device fault parameter description

Name Description
FaultCode Fault code, a string composed of English letters and numbers, where the string represents the fault code in hexadecimal.
FaultLevel

The handling policy corresponding to the highest-level fault among the current faults.

  • NotHandle: No action is taken.
  • SubHealth: Handled according to the configured policy.
  • Reset: Isolate the node.
  • Separate: Isolate the node.
  • RestartRequest: Isolate the node.
UpdateTime ConfigMap update time.
NodeStatus

Current node status.

  • Healthy: The node is healthy.
  • SubHealthy: The node is pre-isolated. The current job is not processed, and subsequent jobs will no longer be scheduled to this node.
  • UnHealthy: The node is unhealthy. Isolate the node and reschedule jobs.

NodeD

NodeD collects node fault information and node health status information, and places it as external information in a K8s ConfigMap for external query and use.

The query command is kubectl describe cm mindx-dl-nodeinfo-<nodename> -n mindx-dl. A command response example is shown below. For key parameter descriptions, see Table 1 mindx-dl-nodeinfo-<nodename>.

Name:         mindx-dl-nodeinfo-<nodename>
Namespace:    mindx-dl
Labels:       <none>
Annotations:  <none>

Data
====
NodeInfo:
----
{"NodeInfo":{"FaultDevList":[{"DeviceType":"CPU","DeviceId":1,"FaultCode":["00000011"],"FaultLevel":"SeparateFault"}],"NodeStatus":"UnHealthy"},"CheckCode":"3a2934c3cb875f2256c770c75a6fdf24594fcf64481ac6cd0d0f74b8fea88855"}
Events:  <none>

Creating an Image

Building a Container Image Using Dockerfile (PyTorch)

Prerequisites

As shown in Table 1, obtain the packages for the corresponding operating system, as well as the Dockerfile and script files required for image packaging.

In the package name, {version} indicates the version, {arch} indicates the architecture, and {chip_type} indicates the chip type. The corresponding CANN packages for versions 6.3.RC3, 6.2.RC3, and later include an installation prompt "Do you accept the EULA to install CANN (Y/N)"; in the Dockerfile writing examples, the installation command includes the --quiet parameter to accept the EULA by default, which you can modify as needed.

Table 1 Required software

Software Package Description Obtaining Method
Ascend-cann-toolkit_{version}_linux-{arch}.run CANN Toolkit.

Download Link

Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run

CANN operator package.

Before CANN 8.5.0, this package was named Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run

 Download Link
apex-0.1+ascend-cp3x-cp3x-linux_{arch}.whl

Mixed precision module.

cp3x in the package name indicates the Python version. For example, x being 10 indicates Python 3.10.

 See the "Installing the APEX Module" section in Ascend Extension for PyTorch Software Installation Guide, and compile the APEX software package based on the actual situation.
  • x86_64: torch-v{version}+cpu-cp3x-cp3x-linux_x86_64.whl
  • ARM: torch-v{version}-cp3x-cp3x-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Official PyTorch package.

cp3x in the package name indicates the Python version. For example, x being 10 indicates Python 3.10.

{version} indicates the PyTorch version. Currently, PyTorch 2.1.0 to 2.7.1 is supported.

Download Link

Select the PyTorch version to install based on the actual situation.

torch_npu-v{version}.post{version}-cp3x-cp3x-manylinux_2_17_{arch}.manylinux2014_{arch}.whl

Ascend Extension for PyTorch plugin.

cp3x in the package name indicates the Python version. For example, x being 10 indicates Python 3.10.

Download Link

  • Select a torch_npu version that is compatible with PyTorch.
  • If using PyTorch models from the MindSpeed-LLM repository, Ascend Extension for PyTorch 2.1.0 or later is required.
Dockerfile Required for creating images. Refer to Dockerfile Writing Example
dllogger-master PyTorch logging tool. Download Link
ascend_install.info Driver installation information file. Copy the "/etc/ascend_install.info" file from the host.
version.info Driver version information file. Copy the "/usr/local/Ascend/driver/version.info" file from the host.
prebuild.sh Performs preparation for training runtime environment installation, such as configuring proxies. Refer to Step 3
install_ascend_pkgs.sh Ascend software package installation script. Refer to Step 4
postbuild.sh Clears installation packages, scripts, proxy configurations, etc., that do not need to be kept in the container. Refer to Step 5

To prevent software packages from being maliciously tampered with during transmission or storage, you need to download the corresponding digital signature file for integrity verification when downloading the software packages.

After downloading the software packages, refer to the OpenPGP Signature Verification Guide to perform PGP digital signature verification on the software packages downloaded from the Support website. If the verification fails, do not use the software package and contact Huawei technical support engineers first.

Before installing or upgrading using a software package, you must also verify the digital signature of the software package following the above process to ensure that the software package has not been tampered with.

For carrier customers, please visit https://support.huawei.com/carrier/digitalSignatureAction.

For enterprise customers, please visit https://support.huawei.com/enterprise/en/tool/pgp-verify-TL1000000054.

Note

This chapter uses the Ubuntu OS with Python 3.10 and CANN 8.5.0 as an example to introduce the detailed process of building a container image using a Dockerfile. You need to modify the relevant steps according to the actual situation during use.

Procedure

  1. Upload the prepared software packages, deep learning framework-related packages, host-side driver installation information file, and driver version information file to the same directory on the server (such as /home/test).

    • Ascend-cann-toolkit_{version}_linux-{arch}.run
    • Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run
    • apex-0.1+ascend-cp310-cp310-linux_{arch}.whl
    • torch-v{version}+cpu.cxx11.abi-cp310-cp310-linux_{arch}.whl or torch-v{version}-cp3x-cp3x-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
    • torch_npu-v{version}.post{version}-cp310-cp310-manylinux_2_17_{arch}.manylinux2014_{arch}.whl
    • dllogger-master
    • ascend_install.info
    • version.info

  2. Log in to the server as the root user.

  3. Perform the following steps to prepare the prebuild.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the prebuild.sh file.

      vi prebuild.sh

    2. For the content to be written, refer to the prebuild.sh writing example. After writing, run the :wq command to save the content. The content uses the Ubuntu OS as an example.

  4. Perform the following steps to prepare the install_ascend_pkgs.sh file.

    1. Go to the directory where software packages are located and run the following command to create the install_ascend_pkgs.sh file.

      vi install_ascend_pkgs.sh

    2. For the content to write, refer to the install_ascend_pkgs.sh writing example. After writing, execute the :wq command to save the content. The content uses the Ubuntu OS as an example.

  5. Perform the following steps to prepare the postbuild.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the postbuild.sh file.

      vi postbuild.sh

    2. For the content to write, refer to the postbuild.sh writing example. After writing, execute the :wq command to save the content. The content uses the Ubuntu OS as an example.

  6. Perform the following steps to prepare a Dockerfile file.

    1. Go to the directory where the software packages are located and run the following command to create a Dockerfile file (file name example: "Dockerfile").

      vi Dockerfile

    2. For the content to be written, refer to the Dockerfile writing example, then run the :wq command to save the content. The content uses the Ubuntu OS as an example.

  7. Go to the directory where the software packages are located and run the following command to build the container image. Note: Do not omit the the dot (".") at the end of the command.

    docker build -t Image name_OS architecture: Image tag .

    In the above command, the description of each parameter is shown in the following table.

    Table 2 Command parameters

    Name Description
    -t Specifies the image name.
    Image name_OS architecture: Image tag Image name and tag. Enter the actual values.

    Example:

    docker build -t test_train_arm64:v1.0 .

    When "Successfully built xxx" appears, it indicates that the image has been built successfully.

  8. After the build is complete, run the following command to view the image information.

    docker images

    The response example is as follows.

    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
test_train_arm64    v1.0                d82746acd7f0        27 minutes ago      749MB

Writing Examples

  1. prebuild.sh

    Ubuntu Arm:

    #!/bin/bash
#--------------------------------------------------------------------------------
# Write script code here using bash syntax for installation preparation, such as configuring proxies
# This script will be executed before the formal build process starts
#
# Note: This script will not be automatically removed after execution. If it does not need to be retained in the image, please remove it in the postbuild.sh script
#--------------------------------------------------------------------------------
# DNS configuration
tee /etc/resolv.conf <<- EOF
nameserver xxx.xxx.xxx.xxx  #DNS server IP, multiple can be specified, configure according to actual setup
nameserver xxx.xxx.xxx.xxx
nameserver xxx.xxx.xxx.xxx
EOF
# apt proxy configuration
tee /etc/apt/apt.conf.d/80proxy <<- EOF
Acquire::http::Proxy "http://xxx.xxx.xxx.xxx:xxx";  # HTTP proxy server IP address and port
Acquire::https::Proxy "http://xxx.xxx.xxx.xxx:xxx";  # HTTPS proxy server IP address and port
EOF
chmod 777 -R /tmp
rm /var/lib/apt/lists/*
# apt source configuration (using Ubuntu 18.04 Arm source as an example; configure according to actual setup)
tee /etc/apt/sources.list <<- EOF
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-backports main restricted universe multiverse
EOF

    Ubuntu x86_64:

    #!/bin/bash
#--------------------------------------------------------------------------------

# Use bash syntax to write script code here to for installation preparation, such as configuring proxies
# This script will be executed before the formal build process starts
#
# Note: This script will not be automatically removed after execution. If it does not need to be retained in the image, please clean it up in the postbuild.sh script
#--------------------------------------------------------------------------------
# apt proxy settings
tee /etc/apt/apt.conf.d/80proxy <<- EOF
Acquire::http::Proxy "http://xxx.xxx.xxx.xxx:xxx";    #HTTP proxy server IP address and port
Acquire::https::Proxy "http://xxx.xxx.xxx.xxx:xxx";   # HTTPS proxy server IP address and port
EOF

#apt source configuration (using Ubuntu 18.04 x86_64 source as an example; configure according to actual setup)
tee /etc/apt/sources.list <<- EOF
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-security main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-security main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main multiverse restricted universe
EOF

  2. install_ascend_pkgs.sh

    #--------------------------------------------------------------------------------
# Use bash syntax to write script code here to install Ascend software packages
#
# Note: This script will not be automatically removed after execution. If it does not need to be retained in the image, please clean it up in the postbuild.sh script
#--------------------------------------------------------------------------------
umask 0022
cp ascend_install.info /etc/
# Before building, copy /usr/local/Ascend/driver/version.info from the host to the current directory
mkdir -p /usr/local/Ascend/driver/
cp version.info /usr/local/Ascend/driver/
# Ascend-cann-toolkit_{version}_linux-{arch}.run
chmod +x Ascend-cann-toolkit_{version}_linux-{arch}.run
chmod +x Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run
./Ascend-cann-toolkit_{version}_linux-{arch}.run --install-path=/usr/local/Ascend/ --install --quiet
echo y | ./Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run --install
# Only installed for the toolkit package, so cleanup is needed; it will be mounted via ascend docker when the container starts
rm -f version.info
rm -rf /usr/local/Ascend/driver/

  3. postbuild.sh

    #--------------------------------------------------------------------------------
# Please use bash syntax to write script code here to clean up installation packages, scripts, proxy configurations, etc. that do not need to be retained in the container
# This script will be executed after the formal build process is completed
#
# Note: This script will be automatically cleared after execution and will not remain in the image; the script location and Working Dir location are /tmp
#--------------------------------------------------------------------------------
rm -f ascend_install.info
rm -f prebuild.sh
rm -f install_ascend_pkgs.sh
rm -f Dockerfile
rm -f Ascend-cann-toolkit_{version}_linux-{arch}.run
rm -f Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run
rm -f apex-0.1+ascend-cp310-cp310-linux_{arch}.whl
rm -f torch-v{version}+cpu.cxx11.abi-cp310-cp310-linux_{arch}.whl
rm -f torch_npu-v{version}.post7-cp310-cp310-manylinux_2_17_{arch}.manylinux2014_{arch}.whl
rm -f /etc/apt/apt.conf.d/80proxy

  4. Dockerfile

    • Ubuntu Arm with Python 3.10:

      FROM ubuntu:18.04
ARG PYTORCH_PKG=torch-v{version}+cpu.cxx11.abi-cp310-cp310-linux_aarch64.whl
ARG PYTORCH_NPU_PKG=torch_npu-v{version}.post{version}-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
ARG APEX_PKG=apex-0.1_ascend-cp310-cp310-linux_aarch64.whl
ARG HOST_ASCEND_BASE=/usr/local/Ascend
ARG TOOLKIT_PATH=/usr/local/Ascend/cann
ARG INSTALL_ASCEND_PKGS_SH=install_ascend_pkgs.sh
ARG PREBUILD_SH=prebuild.sh
ARG POSTBUILD_SH=postbuild.sh
WORKDIR /tmp
COPY . ./
# Trigger prebuild.sh
RUN bash -c "test -f $PREBUILD_SH && bash $PREBUILD_SH || true"
ENV http_proxy http://xxx.xxx.xxx.xxx:xxx
ENV https_proxy http://xxx.xxx.xxx.xxx:xxx
# System packages
RUN apt update && \
    apt install -y --no-install-recommends curl g++ pkg-config unzip wget build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \
        libblas3 liblapack3 liblapack-dev openssl libssl-dev libblas-dev gfortran libhdf5-dev libffi-dev libicu60 libxml2 \
        patch libbz2-dev llvm libncursesw5-dev xz-utils liblzma-dev m4 dos2unix libopenblas-dev libsqlite3-dev
RUN wget https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tgz
RUN tar -zxvf Python-3.10.5.tgz && cd Python-3.10.5 && ./configure --prefix=/usr/local/python3.10.5 --enable-shared && make && make install
RUN ln -s /usr/local/python3.10.5/bin/python3.10 /usr/local/python3.10.5/bin/python && \
    ln -s /usr/local/python3.10.5/bin/pip3.10 /usr/local/python3.10.5/bin/pip
# Configure Python pip source
RUN mkdir -p ~/.pip \
&& echo '[global] \n\
index-url=https://pypi.doubanio.com/simple/\n\
trusted-host=pypi.doubanio.com' >> ~/.pip/pip.conf

ENV LD_LIBRARY_PATH=/usr/local/python3.10.5/lib:$LD_LIBRARY_PATH
ENV PATH=/usr/local/python3.10.5/bin:$PATH
ENV PYTHONPATH=/usr/local/python3.10.5/lib/python3.10/site-packages:$PYTHONPATH
# Python packages
RUN pip3 install decorator && \
    pip3 install sympy && \
    pip3 install cffi && \
    pip3 install pyyaml && \
    pip3 install pathlib2 && \
    pip3 install grpcio && \
    pip3 install grpcio-tools && \
    pip3 install protobuf && \
    pip3 install scipy && \
    pip3 install requests && \
    pip3 install attrs && \
    pip3 install Pillow==9.1.0 && \
    pip3 install torchvision==0.16.0 && \
    pip3 install numpy==1.23.5 && \
    pip3 install psutil && \
    pip3 install absl-py

# Create the HwHiAiUser user and owner. Ensure the UID and GID are consistent with the physical machine to avoid ownerless files. In this example, the user and corresponding group are automatically created, with both UID and GID set to 1000
RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser
# Ascend packages
RUN umask 0022 && bash $INSTALL_ASCEND_PKGS_SH
RUN umask 0022 && pip3 install $APEX_PKG
RUN umask 0022 && pip3 install $PYTORCH_PKG
RUN umask 0022 && pip3 install $PYTORCH_NPU_PKG
RUN cd /tmp/dllogger-master/ && \
    python3 setup.py build && \
    python3 setup.py install
# Environment variables
ENV HCCL_WHITELIST_DISABLE=1
ENV PYTHONPATH=/tmp/dllogger-master
# Create /lib64/ld-linux-aarch64.so.1
RUN umask 0022 && \
    if [ ! -d "/lib64" ]; \
    then \
        mkdir /lib64 && ln -sf /lib/ld-linux-aarch64.so.1 /lib64/ld-linux-aarch64.so.1; \
    fi
ENV http_proxy ""
ENV https_proxy ""
# Trigger postbuild.sh
RUN bash -c "test -f $POSTBUILD_SH && bash $POSTBUILD_SH || true" && \
    rm $POSTBUILD_SH

    • Ubuntu x86_64 with Python 3.10:

      FROM ubuntu:18.04
ARG PYTORCH_PKG=torch-v{version}+cpu.cxx11.abi-cp310-cp310-linux_x86_64.whl
ARG PYTORCH_NPU_PKG=torch_npu-v{version}.post{version}-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
ARG APEX_PKG=apex-0.1_ascend-cp310-cp310-linux_x86_64.whl
ARG HOST_ASCEND_BASE=/usr/local/Ascend
ARG TOOLKIT_PATH=/usr/local/Ascend/cann
ARG INSTALL_ASCEND_PKGS_SH=install_ascend_pkgs.sh
ARG PREBUILD_SH=prebuild.sh
ARG POSTBUILD_SH=postbuild.sh
WORKDIR /tmp
COPY . ./
# Trigger prebuild.sh
RUN bash -c "test -f $PREBUILD_SH && bash $PREBUILD_SH || true"
ENV http_proxy http://xxx.xxx.xxx.xxx:xxx
ENV https_proxy http://xxx.xxx.xxx.xxx:xxx
# System packages
RUN apt update && \
    apt install -y --no-install-recommends curl g++ pkg-config unzip wget build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \
        libblas3 liblapack3 liblapack-dev openssl libssl-dev libblas-dev gfortran libhdf5-dev libffi-dev libicu60 libxml2 \
        patch libbz2-dev llvm libncursesw5-dev xz-utils liblzma-dev m4 dos2unix libopenblas-dev libsqlite3-dev
RUN wget https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tgz
RUN tar -zxvf Python-3.10.5.tgz && cd Python-3.10.5 && ./configure --prefix=/usr/local/python3.10.5 --enable-shared && make && make install
RUN ln -s /usr/local/python3.10.5/bin/python3.10 /usr/local/python3.10.5/bin/python && \
    ln -s /usr/local/python3.10.5/bin/pip3.10 /usr/local/python3.10.5/bin/pip
# Configure Python pip source
RUN mkdir -p ~/.pip \
&& echo '[global] \n\
index-url=https://pypi.doubanio.com/simple/\n\
trusted-host=pypi.doubanio.com' >> ~/.pip/pip.conf

ENV LD_LIBRARY_PATH=/usr/local/python3.10.5/lib:$LD_LIBRARY_PATH
ENV PATH=/usr/local/python3.10.5/bin:$PATH
ENV PYTHONPATH=/usr/local/python3.10.5/lib/python3.10/site-packages:$PYTHONPATH
# Python packages
RUN pip3 install decorator && \
    pip3 install sympy && \
    pip3 install cffi && \
    pip3 install pyyaml && \
    pip3 install pathlib2 && \
    pip3 install grpcio && \
    pip3 install grpcio-tools && \
    pip3 install protobuf && \
    pip3 install scipy && \
    pip3 install requests && \
    pip3 install attrs && \
    pip3 install Pillow==9.1.0 && \
    pip3 install torchvision==0.16.0 && \
    pip3 install numpy==1.23.5 && \
    pip3 install psutil && \
    pip3 install absl-py

# Create the HwHiAiUser user and owner. Ensure the UID and GID are consistent with the physical machine to avoid files without an owner. In this example, the user and corresponding group are automatically created, with both UID and GID set to 1000
RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser
# Ascend packages
RUN bash $INSTALL_ASCEND_PKGS_SH
RUN pip3 install $APEX_PKG
RUN pip3 install $PYTORCH_PKG
RUN pip3 install $PYTORCH_NPU_PKG
RUN cd /tmp/dllogger-master/ && \
    python3 setup.py build && \
    python3 setup.py install
# Environment variables
ENV HCCL_WHITELIST_DISABLE=1
ENV PYTHONPATH=/tmp/dllogger-master
ENV http_proxy ""
ENV https_proxy ""
# Trigger postbuild.sh
RUN bash -c "test -f $POSTBUILD_SH && bash $POSTBUILD_SH || true" && \
    rm $POSTBUILD_SH

Building a Container Image Using a Dockerfile (MindSpore)

Prerequisites

As shown in Table 1, obtain the software packages for the corresponding OS, as well as the Dockerfile and script files required for packaging the image.

In the package name, {version} indicates the version number, {arch} indicates the architecture, and {chip_type} indicates the chip type. The corresponding CANN packages in version 6.3.RC3, 6.2.RC3, and later include an installation prompt "Do you accept the EULA to install CANN (Y/N)"; in the Dockerfile writing examples, the installation command includes the "--quiet" parameter to accept the EULA by default, which users can modify as needed.

Note

The MindSpore package and software used in Atlas training series products must meet the corresponding compatibility requirements. See MindSpore Installation Guide for details.

Table 1 Required software

Software Package Description Obtaining Method
Ascend-cann-toolkit_{version}_linux-{arch}.run CANN Toolkit.

Download Link

A version earlier than CANN 8.5.0 is required.
Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run CANN operator package.

Download Link

A version earlier than CANN 8.5.0 is required.
mindspore-{version}-cp3x-cp3x-linux_{arch}.whl

MindSpore framework whl package.

Currently supports Python 3.9 to 3.11. The x in the package name indicates 9, 10, or 11. Select the corresponding software package based on the actual situation.

For versions earlier than MindSpore 2.0.0, the package name is changed from mindspore to mindspore-ascend.

 Download Link
Dockerfile Required for creating an image. Refer to Dockerfile writing example.
ascend_install.info Driver installation information file. Copy the "/etc/ascend_install.info" file from the host.
version.info Driver version information file. Copy the "/usr/local/Ascend/driver/version.info" file from the host.
prebuild.sh Performs preparation for training runtime environment installation, such as configuring proxies. Refer to Step 3.
install_ascend_pkgs.sh Ascend software package installation script. Refer to Step 4.
postbuild.sh Clears installation packages, scripts, proxy configurations, and other items that do not need to be retained in the container. Refer to Step 5.

To prevent software packages from being maliciously tampered with during transmission or storage, you need to download the corresponding digital signature file for integrity verification when downloading the software package.

After downloading the software package, refer to OpenPGP Signature Verification Guide to perform PGP digital signature verification on the software package downloaded from the Support website. If the verification fails, do not use the software package and contact Huawei technical support engineers for resolution first.

Before installing or upgrading using a software package, you must also verify the digital signature of the software package as described above to ensure that it has not been tampered with.

For carrier customers, please visit https://support.huawei.com/carrier/digitalSignatureAction.

For enterprise customers, please visit https://support.huawei.com/enterprise/en/tool/pgp-verify-TL1000000054

Note

  • This section uses Ubuntu 18.04 with Python 3.9 as an example to describe the detailed process of building a container image using a Dockerfile. You need to modify the relevant steps based on the actual situation.
  • If you are using MindSpore 2.0.3 or later, you need to use ubuntu:20.04.

Procedure

  1. Upload the prepared software packages, deep learning framework, host-side driver installation information file, and driver version information file to the same directory on the server (such as /home/test).

    • Ascend-cann-toolkit_{version}_linux-{arch}.run
    • Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run
    • mindspore-{version}-cp3x-cp3x-linux_{arch}.whl
    • ascend_install.info
    • version.info

  2. Log in to the server as the root user.

  3. Perform the following steps to prepare the prebuild.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the prebuild.sh file.

      vi prebuild.sh

    2. For the content to write, see the prebuild.sh writing example. After writing, run the :wq command to save the content. The content uses the Ubuntu OS as an example.

  4. Perform the following steps to prepare the install_ascend_pkgs.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the install_ascend_pkgs.sh file.

      vi install_ascend_pkgs.sh

    2. For the content to write, see the install_ascend_pkgs.sh writing example. After writing, run the :wq command to save the content. The content uses the Ubuntu OS as an example.

  5. Perform the following steps to prepare the postbuild.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the postbuild.sh file.

      vi postbuild.sh

    2. For the content to write, refer to the postbuild.sh writing example. After writing, execute the :wq command to save the content. The content uses the Ubuntu OS as an example.

  6. Perform the following steps to prepare a Dockerfile.

    1. Go to the directory where the software package is located and run the following command to create a Dockerfile (file name example: "Dockerfile").

      vi Dockerfile

    2. For the content to write, refer to the Dockerfile writing example. After writing, execute the :wq command to save the content. The content uses the Ubuntu OS as an example.

  7. Go to the directory where the software packages are located and run the following command to build the container image. Note: Do not omit the dot (".") at the end of the command.

    docker build -t image_name_system_architecture:image_tag .

In the above command, the parameters are described in the following table.

Table 2 Command parameter description

|Name|Description|
|--|--|
|-t|Image name|
|image_name_system_architecture:image_tag|Image name and tag. Use the actual values.|

For example:

```shell
docker build -t test_train_arm64:v1.0 .
```

When "Successfully built xxx" appears, it indicates that the image has been built successfully.

  1. After the build is complete, run the following command to view the image information.

    docker images

    The response example is as follows:

    REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
test_train_arm64          v1.0                d82746acd7f0        27 minutes ago      749MB

  2. (Optional) Verify that the base image is available.

    1. Use Ascend Docker Runtime to mount the driver in the base image, using the base image test_train_arm64:v1.0 as an example.

      docker run -it --privileged -e ASCEND_VISIBLE_DEVICES=0 test_train_arm64:v1.0 /bin/bash

    2. Check whether the MindSpore software in the base image is installed successfully.

      python -c "import mindspore;mindspore.set_context(device_target='Ascend');mindspore.run_check()"

      The response example is as follows, indicating that MindSpore is installed successfully.

      MindSpore version: Version number
The result of multiplication calculation is correct, MindSpore has been installed on platform [Ascend] successfully!

Writing Examples

  1. prebuild.sh

    • Ubuntu Arm:

      #!/bin/bash
#--------------------------------------------------------------------------------
# # Write script code here using bash syntax to for installation preparations, such as configuring proxies
# # This script will be executed before the formal build process starts
#
# # Note: This script will not be automatically cleaned up after execution. If it does not need to be retained in the image, please clean it up in the postbuild.sh script
#--------------------------------------------------------------------------------
# # DNS settings
tee /etc/resolv.conf <<- EOF
nameserver xxx.xxx.xxx.xxx  ## DNS server IP, multiple can be filled in, configure according to actual conditions
nameserver xxx.xxx.xxx.xxx
nameserver xxx.xxx.xxx.xxx
EOF
# apt proxy settings
tee /etc/apt/apt.conf.d/80proxy <<- EOF
Acquire::http::Proxy "http://xxx.xxx.xxx.xxx:xxx";  #HTTP proxy server IP address and port.
Acquire::https::Proxy "http://xxx.xxx.xxx.xxx:xxx";  #HTTPS proxy server IP address and port.
EOF
chmod 777 -R /tmp
rm /var/lib/apt/lists/*
#apt source settings (using Ubuntu 18.04 ARM source as an example, configure according to actual setup)
tee /etc/apt/sources.list <<- EOF
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu-ports/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu-ports/ bionic-backports main restricted universe multiverse
EOF

    • Ubuntu x86_64:

      #!/bin/bash
#--------------------------------------------------------------------------------

# # Write script code here using bash syntax for installation preparations, such as configuring proxies
# # This script will be executed before the formal build process starts
#
# # Note: This script will not be automatically removed after execution. If it does not need to be retained in the image, please remove it in the postbuild.sh script
#--------------------------------------------------------------------------------
# # apt proxy settings
tee /etc/apt/apt.conf.d/80proxy <<- EOF
Acquire::http::Proxy "http://xxx.xxx.xxx.xxx:xxx";    ## HTTP proxy server IP address and port
Acquire::https::Proxy "http://xxx.xxx.xxx.xxx:xxx";   #HTTPS proxy server IP address and port
EOF

#apt source configuration (using Ubuntu 18.04 x86_64 source as an example, configure according to actual setup)
tee /etc/apt/sources.list <<- EOF
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-security main multiverse restricted universe
deb http://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-security main multiverse restricted universe
deb-src http://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main multiverse restricted universe
EOF

  2. install_ascend_pkgs.sh

    #!/bin/bash
#--------------------------------------------------------------------------------
# Use bash syntax to write script code here to install Ascend software packages
#
# Note: This script will not be automatically removed after execution. If it does not need to be retained in the image, please clean it up in the postbuild.sh script
#--------------------------------------------------------------------------------
# Before building, copy /etc/ascend_install.info from the host to the current directory
cp ascend_install.info /etc/
mkdir -p /usr/local/Ascend/driver/
cp version.info /usr/local/Ascend/driver/

# Ascend-cann-toolkit_{version}_linux-{arch}.run
chmod +x Ascend-cann-toolkit_{version}_linux-{arch}.run
./Ascend-cann-toolkit_{version}_linux-{arch}.run --install-path=/usr/local/Ascend/ --install --quiet
chmod +x Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run
./Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run --install --quiet

# Only install the toolkit package, which needs to be cleaned up. It will be mounted via ascend docker when the container starts
rm -f version.info
rm -rf /usr/local/Ascend/driver/

  3. postbuild.sh

    #!/bin/bash
#--------------------------------------------------------------------------------
# Please write script code here using bash syntax to clean up installation packages, scripts, proxy configurations, etc. that do not need to be retained in the container
# This script will be executed after the formal build process is completed
#
# Note: This script will be automatically cleared after execution and will not remain in the image. The script location and Working Dir location are /root.
#--------------------------------------------------------------------------------

rm -f ascend_install.info
rm -f prebuild.sh
rm -f install_ascend_pkgs.sh
rm -f Dockerfile
rm -f version.info
rm -f Ascend-cann-toolkit_{version}_linux-{arch}.run
rm -f Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run
# Select the packages to delete based on the actual installed version.
rm -f mindspore-{version}-cp3x-cp3x-linux_{arch}.whl
rm -f /etc/apt/apt.conf.d/80proxy

tee /etc/resolv.conf <<- EOF
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

options edns0

nameserver xxx.xxx.xxx.xxx
nameserver xxx.xxx.xxx.xxx
EOF

  4. Dockerfile

    • Ubuntu Arm system with Python 3.9:

      FROM ubuntu:18.04

ARG HOST_ASCEND_BASE=/usr/local/Ascend
ARG INSTALL_ASCEND_PKGS_SH=install_ascend_pkgs.sh
ARG TOOLKIT_PATH=/usr/local/Ascend/ascend-toolkit/latest
ARG MINDSPORE_PKG=mindspore-{version}-cp39-cp39-linux_aarch64.whl
ARG PREBUILD_SH=prebuild.sh
ARG POSTBUILD_SH=postbuild.sh
WORKDIR /tmp
COPY . ./

# Trigger prebuild.sh
RUN bash -c "test -f $PREBUILD_SH && bash $PREBUILD_SH"

ENV http_proxy http://xxx
ENV https_proxy http://xxx


# # system packages
RUN apt update && \
    apt install --no-install-recommends curl g++ pkg-config unzip wget build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \
        libblas3 liblapack3 liblapack-dev openssl libssl-dev libblas-dev gfortran libhdf5-dev libffi-dev libicu60 libxml2 -y

RUN wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz
RUN tar -zxvf Python-3.9.2.tgz && cd Python-3.9.2 && ./configure --prefix=/usr/local/python3.9.2 --enable-shared && make && make install

RUN ln -s /usr/local/python3.9.2/bin/python3.9 /usr/local/python3.9.2/bin/python && \
    ln -s /usr/local/python3.9.2/bin/pip3.9 /usr/local/python3.9.2/bin/pip


# # configure Python pip source
RUN mkdir -p ~/.pip \
&& echo '[global] \n\
index-url=https://pypi.doubanio.com/simple/\n\
trusted-host=pypi.doubanio.com' >> ~/.pip/pip.conf

# # users need to modify the PYTHONPATH path according to the actual situation
ENV LD_LIBRARY_PATH=/usr/local/python3.9.2/lib:$LD_LIBRARY_PATH
ENV PATH=/usr/local/python3.9.2/bin:$PATH
ENV PYTHONPATH=/usr/local/python3.9.2/lib/python3.9/site-packages:$PYTHONPATH
# # create HwHiAiUser user and owner. UID and GID must be consistent with the physical machine to avoid ownerless files. The example will automatically create the user and corresponding group, with both UID and GID set to 1000
RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser

# # install Python 3.9. If installing other versions, modify the following commands according to the actual situation
RUN pip install numpy && \
    pip install decorator && \
    pip install sympy==1.4 && \
    pip install cffi==1.12.3 && \
    pip install pyyaml && \
    pip install pathlib2 && \
    pip install grpcio && \
    pip install grpcio-tools && \
    pip install protobuf && \
    pip install scipy && \
    pip install requests && \
    pip install kubernetes && \
    pip install attrs && \
    pip install psutil && \
    pip install absl-py

# Ascend packages
RUN umask 0022 && bash $INSTALL_ASCEND_PKGS_SH

# MindSpore installation
RUN pip install $MINDSPORE_PKG

ENV http_proxy ""
ENV https_proxy ""

# Trigger postbuild.sh
RUN bash -c "test -f $POSTBUILD_SH && bash $POSTBUILD_SH" && \
    rm $POSTBUILD_SH

    • Ubuntu x86_64 with Python 3.9:

      FROM ubuntu:18.04

ARG HOST_ASCEND_BASE=/usr/local/Ascend
ARG INSTALL_ASCEND_PKGS_SH=install_ascend_pkgs.sh
ARG TOOLKIT_PATH=/usr/local/Ascend/ascend-toolkit/latest
ARG MINDSPORE_PKG=mindspore-{version}-cp39-cp39-linux_x86_64.whl
ARG PREBUILD_SH=prebuild.sh
ARG POSTBUILD_SH=postbuild.sh
WORKDIR /tmp
COPY . ./

# Trigger prebuild.sh
RUN bash -c "test -f $PREBUILD_SH && bash $PREBUILD_SH"

ENV http_proxy http://xxx
ENV https_proxy http://xxx


# # System packages
RUN apt update && \
    apt install --no-install-recommends curl g++ pkg-config unzip wget build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev \
        libblas3 liblapack3 liblapack-dev openssl libssl-dev libblas-dev gfortran libhdf5-dev libffi-dev libicu60 libxml2 -y

RUN wget https://www.python.org/ftp/python/3.9.2/Python-3.9.2.tgz
RUN tar -zxvf Python-3.9.2.tgz && cd Python-3.9.2 && ./configure --prefix=/usr/local/python3.9.2 --enable-shared && make && make install

RUN ln -s /usr/local/python3.9.2/bin/python3.9 /usr/local/python3.9.2/bin/python && \
    ln -s /usr/local/python3.9.2/bin/pip3.9 /usr/local/python3.9.2/bin/pip

# # Configure Python pip source
RUN mkdir -p ~/.pip \
&& echo '[global] \n\
index-url=https://pypi.doubanio.com/simple/\n\
trusted-host=pypi.doubanio.com' >> ~/.pip/pip.conf

# # Modify the PYTHONPATH path according to the actual situation
ENV LD_LIBRARY_PATH=/usr/local/python3.9.2/lib:$LD_LIBRARY_PATH
ENV PATH=/usr/local/python3.9.2/bin:$PATH
ENV PYTHONPATH=/usr/local/python3.9.2/lib/python3.9/site-packages:$PYTHONPATH
# # Create HwHiAiUser user and owner. UID and GID should be consistent with the physical machine to avoid ownerless files. In this example, the user and corresponding group are automatically created, with both UID and GID set to 1000
RUN useradd -d /home/HwHiAiUser -u 1000 -m -s /bin/bash HwHiAiUser

# # Install Python 3.9. If installing other versions, modify the following commands according to the actual situation
RUN pip install numpy && \
    pip install decorator && \
    pip install sympy==1.4 && \
    pip install cffi==1.12.3 && \
    pip install pyyaml && \
    pip install pathlib2 && \
    pip install grpcio && \
    pip install grpcio-tools && \
    pip install protobuf && \
    pip install scipy && \
    pip install requests && \
    pip install kubernetes && \
    pip install attrs && \
    pip install psutil && \
    pip install absl-py

# Ascend packages
RUN umask 0022 && bash $INSTALL_ASCEND_PKGS_SH

# MindSpore installation
RUN pip install $MINDSPORE_PKG

ENV http_proxy ""
ENV https_proxy ""

# Trigger postbuild.sh
RUN bash -c "test -f $POSTBUILD_SH && bash $POSTBUILD_SH" && \
    rm $POSTBUILD_SH

Building an inference image using a Dockerfile

Prerequisites

Obtain the software packages for the corresponding operating system, as well as the Dockerfile and script files required for building the image, as shown in Table 1.

In the package name, {version} indicates the version number, {arch} indicates the architecture, and {chip_type} indicates the chip type. For the matching CANN software packages at version 6.3.RC3, 6.2.RC3, and later, an installation prompt "Do you accept the EULA to install CANN (Y/N)" has been added. In the Dockerfile writing examples, the installation command includes the --quiet parameter to accept the EULA by default, which users can modify as needed.

Table 1 Required software

Software Package Description Obtaining Method
Ascend-cann-toolkit_{version}_linux-{arch}.run CANN Toolkit Download Link
Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run

CANN operator package.

Before CANN 8.5.0, this package was named Ascend-cann-kernels-{chip_type}_{version}_linux-{arch}.run

 Download Link
Dockerfile Required for creating images. Refer to Dockerfile Writing Example.
install.sh Script for installing the inference service. For creating inference models, refer to ResNet50 Inference Guide.
XXX.tar Name of the inference service code package, prepared by the user based on the inference service. This chapter uses dvpp_resnet.tar as an example. For creating inference models, refer to ResNet50 Inference Guide.
run.sh Script for starting the inference service. For creating inference models, refer to ResNet50 Inference Guide.

Note

You need to prepare other software packages and code required for inference on your own.

To prevent software packages from being maliciously tampered with during transmission or storage, you need to download the corresponding digital signature file for integrity verification when downloading software packages.

After downloading the software package, refer to the OpenPGP Signature Verification Guide to perform PGP digital signature verification on the software package downloaded from the Support website. If the verification fails, do not use the software package and contact Huawei technical support engineers for resolution first.

Before using a software package for installation or upgrade, you also need to verify the digital signature of the software package following the above process to ensure that the software package has not been tampered with.

For carrier customers, please visit https://support.huawei.com/carrier/digitalSignatureAction.

For enterprise customers, please visit https://support.huawei.com/enterprise/en/tool/pgp-verify-TL1000000054.

This section uses Ubuntu x86_64 as an example. The code in the following steps is sample code. You can customize it based on the examples, and it is recommended that you perform security hardening on the sample code and images. Refer to Container Security Hardening.

Procedure

  1. Upload the prepared software packages and files to the same directory on the server (for example, /home/infer).

    • Ascend-cann-toolkit_{version}_linux-{arch}.run
    • Ascend-cann-{chip_type}-ops__{version}__linux-{arch}.run
    • Dockerfile
    • install.sh
    • run.sh
    • XXX.tar (self-prepared inference code or script)

  2. Log in to the server as the root user.

  3. Perform the following steps to prepare the install.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the install.sh file.

      vi install.sh

    2. Refer to the install.sh writing example and write the file based on your actual service requirements. After writing, run the :wq command to save the content.

  4. Perform the following steps to prepare the run.sh file.

    1. Go to the directory where the software packages are located and run the following command to create the run.sh file.

      vi run.sh

    2. Refer to the run.sh example and write the script based on your actual service requirements. After writing, run the :wq command to save the content.

  5. Perform the following steps to prepare a Dockerfile.

    1. Go to the directory where the software package is located and run the following command to create a Dockerfile (example filename: "Dockerfile").

      vi Dockerfile

    2. Refer to the Dockerfile example and write the file based on your actual service requirements. After writing, run the :wq command to save the content.

  6. Go to the directory where the software packages are located and run the following command to build the container image. Be careful not to omit the dot (".") at the end of the command.

    docker build --build-arg TOOLKIT_VERSION={version} --build-arg TOOLKIT_ARCH={arch} --build-arg DIST_PKG=XXX.tar -t image_name_system_architecture:image_tag .

    In the above commands, the description of each parameter is shown in the following table.

    Table 2 Command parameters

    Name Description
    --build-arg Parameters within the Dockerfile.
    {version} Toolkit package version number. Enter the actual value.
    {arch} Toolkit package architecture. Enter the actual value based on your situation.
    XXX.tar Name of the inference service code package. Enter the actual value based on your situation.
    -t Image name.
    Image name_system architecture:Image tag Image name and tag. Enter the actual values based on your situation.

    An example is as follows:

    docker build --build-arg TOOLKIT_VERSION=20.1.rc3 --build-arg TOOLKIT_ARCH=x86_64 --build-arg DIST_PKG=dvpp_resnet.tar -t ubuntu-infer:v1 .

    "Successfully built xxx" indicates that the image has been built successfully.

  7. After the build is complete, run the following command to view the image information.

    docker images

    The response example is as follows:

    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
ubuntu-infer        v1                  fffbd83be42a        2 minutes ago       293MB

Writing Examples

  1. install.sh

    #!/bin/bash
#--------------------------------------------------------------------------------
# Install the inference service script. This example uses the inference service package dvpp_resnet.tar for illustration. You can modify the service package name as needed.
#-------------------------------------
tar -xvf dvpp_resnet.tar
# It is also recommended to modify the permissions and owner of the extracted files.

  2. run.sh writing

    #!/bin/bash
# Run service code
cd /home/out
numbers=`ls /dev/| grep davinci | grep -v davinci_manager | wc -l`
# Update logs every 5 minutes
#./main $numbers|grep -nE '.*\[.*[[:digit:]]{2}:[[:digit:]]{1}[05]:00\]' >./log.txt
# Load offline inference environment variables
export LD_LIBRARY_PATH=/usr/local/Ascend/driver/lib64/common:/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/driver/lib64:${LD_LIBRARY_PATH}
source /usr/local/Ascend/cann/set_env.sh
./main $numbers

    Note

    The driver-related paths are configured in LD_LIBRARY_PATH, and the files within them will be used when executing inference jobs. It is recommended that the running user for inference jobs be consistent with the running user specified during driver installation to avoid privilege escalation risks caused by user mismatch.

  3. Dockerfile ( Please customize and modify it according to the actual situation.)

    #The base image ubuntu:18.04 does not include the Toolkit package. You can refer to some steps in the Dockerfile example for installation, and you need to prepare the Toolkit package in advance.
#It is recommended to pull the inference base image from the Ascend image repository, which already has the Toolkit package installed. Also, confirm whether the Toolkit package matches the driver version on the physical machine.
FROM ubuntu:18.04

# Set Toolkit and OPS package parameters
ARG TOOLKIT_VERSION
ARG TOOLKIT_ARCH
ARG TOOLKIT_PKG=Ascend-cann-toolkit_{version}_linux-{arch}.run
ARG OPS_PKG=Ascend-cann-{chip_type}-ops_{version}_linux-{arch}.run


# Set environment variables
ARG ASCEND_BASE=/usr/local/Ascend

# Set the working directory for the container after startup
WORKDIR /home

## Copy the Toolkit package and OPS package
COPY $TOOLKIT_PKG .
COPY $OPS_PKG .

# # Install the Toolkit package and OPS package
RUN umask 0022 && \
    groupadd xxx (User-defined; be consistent with that specified during driver installation) && \
    useradd -g xxx(User-defined; be consistent with that specified during driver installation)） -s /usr/sbin/nologin (user login disabling; Ubuntu as an example) -m -d /home/xxx xxx(User-defined; be consistent with that specified during driver installation) && \
    chmod +x ${TOOLKIT_PKG} &&\
    ./${TOOLKIT_PKG} --quiet --install --install-for-all --whitelist=nnrt --force &&\
    rm ${TOOLKIT_PKG}
    chmod +x ${OPS_PKG} &&\
    ./${OPS_PKG} --install --install-for-all --quiet --force &&\
    rm ${OPS_PKG}

# # Copy the compressed package of the service inference program, installation script, and running script
ARG DIST_PKG
COPY $DIST_PKG .
COPY install.sh .
COPY run.sh .

# # Run the installation script
RUN mkdir -p /usr/slog && \
    mkdir -p /var/log/npu/slog/slogd && \
    chmod u+x run.sh install.sh && \
    sh install.sh && \
    rm $DIST_PKG && \
    rm install.sh

CMD bash run.sh

    Note

    For CANN package versions 6.2.RC1, 6.3.RC1, and later, the --force parameter is added when installing the package. This parameter has already been included in the Dockerfile example above. If you are using a package version earlier than 6.2.RC1 or 6.3.RC1, you need to remove this parameter from the Dockerfile example.

Quering Information About Currently Available Devices in the Cluster

  1. Query the ConfigMap.

    kubectl get cm -A | grep cluster-info

    The response example is as follows:

    kube-public            cluster-info                                           1      19d
mindx-dl               cluster-info-device-0                                  1      19h
mindx-dl               cluster-info-node-cm                                   1      19h
mindx-dl               cluster-info-switch-0                                  1      19h

  2. Query the detailed information of the ConfigMap to obtain available device information. The following uses the node name localhost.localdomain as an example.

    1. Query the detailed information of the device-related ConfigMap to obtain the available chip information of the node.

      kubectl describe cm -n mindx-dl cluster-info-device-0

      The response example is as follows:

      Name:         cluster-info-device-0
Namespace:    mindx-dl
Labels:       mx-consumer-volcano=true
Annotations:  <none>
Data
====
cluster-info-device-0:
----
{"mindx-dl-deviceinfo-localhost.localdomain":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-3,Ascend910-4,Ascend910-5,Ascend910-6,Ascend910-7","huawei.com/Ascend910-Fault":"[{\"fault_type\":\"PublicFault\",\"npu_name\":\"Ascend910-0\",\"large_model_fault_level\":\"SeparateNPU\",\"fault_level\":\"SeparateNPU\",\"fault_handling\":\"SeparateNPU\",\"fault_code\":\"220001001\",\"fault_time_and_level_map\":{\"220001001\":{\"fault_time\":1736926605,\"fault_level\":\"SeparateNPU\"}}},{\"fault_type\":\"PublicFault\",\"npu_name\":\"Ascend910-1\",\"large_model_fault_level\":\"SeparateNPU\",\"fault_level\":\"SeparateNPU\",\"fault_handling\":\"SeparateNPU\",\"fault_code\":\"220001001\",\"fault_time_and_level_map\":{\"220001001\":{\"fault_time\":1736926605,\"fault_level\":\"SeparateNPU\"}}},{\"fault_type\":\"PublicFault\",\"npu_name\":\"Ascend910-2\",\"large_model_fault_level\":\"SeparateNPU\",\"fault_level\":\"SeparateNPU\",\"fault_handling\":\"SeparateNPU\",\"fault_code\":\"220001001\",\"fault_time_and_level_map\":{\"220001001\":{\"fault_time\":1736926605,\"fault_level\":\"SeparateNPU\"}}}]","huawei.com/Ascend910-NetworkUnhealthy":"","huawei.com/Ascend910-Recovering":"","huawei.com/Ascend910-Unhealthy":"Ascend910-0,Ascend910-1,Ascend910-2"},"UpdateTime":1759214666,"CmName":"mindx-dl-deviceinfo-localhost.localdomain","SuperPodID":-2,"ServerIndex":-2},"mindx-dl-deviceinfo-node173":{"DeviceList":{"huawei.com/Ascend910":"Ascend910-2,Ascend910-3,Ascend910-4,Ascend910-5,Ascend910-6,Ascend910-7","huawei.com/Ascend910-Fault":"[]","huawei.com/Ascend910-NetworkUnhealthy":"","huawei.com/Ascend910-Recovering":"","huawei.com/Ascend910-Unhealthy":""},"UpdateTime":1759202968,"CmName":"mindx-dl-deviceinfo-node173","SuperPodID":-2,"ServerIndex":-2}}
Events:  <none>

      From the above response information, you can see that the available chips for this node are Ascend910-3, Ascend910-4, Ascend910-5, Ascend910-6, and Ascend910-7.

    2. Query the detailed information of the node-related ConfigMap to obtain node status information.

      kubectl describe cm -n mindx-dl cluster-info-node-cm

      The response example is as follows:

      Name:         cluster-info-node-cm
Namespace:    mindx-dl
Labels:       mx-consumer-volcano=true
Annotations:  <none>

Data
====
cluster-info-node-cm:
----
{"mindx-dl-nodeinfo- localhost.localdomain":{"FaultDevList":[{"DeviceType":"PSU","DeviceId":4,"FaultCode":["0300000D"],"FaultLevel":"NotHandleFault"}],"NodeStatus":"Healthy","CmName":"mindx-dl-nodeinfo-localhost.localdomain "}}

BinaryData
====

Events:  <none>

      From the above response information, you can see that NodeStatus of this node is Healthy, indicating that the current node health is normal.

    3. Query the detailed information of the switch-related ConfigMap to obtain node status information.

      kubectl describe cm -n mindx-dl cluster-info-switch-0

      The response example is as follows:

      Name:         cluster-info-switch-0
Namespace:    mindx-dl
Labels:       mx-consumer-volcano=true
Annotations:  <none>

Data
====
cluster-info-switch-0:
----
{"mindx-dl-switchinfo-localhost.localdomain ":{"FaultCode":[],"FaultLevel":"","UpdateTime":1763544679,"NodeStatus":"Healthy","FaultTimeAndLevelMap":{},"CmName":"mindx-dl-switchinfo-localhost.localdomain "}}

BinaryData
====

Events:  <none>

      From the above response information, you can see that NodeStatus of this node is Healthy, indicating that the current node is healthy.

    Based on the above query results, the available chips for this node are Ascend910-3, Ascend910-4, Ascend910-5, Ascend910-6, and Ascend910-7.

    If NodeStatus in the response information of step 2 or step 3 is UnHealthy, it indicates that all devices on the current node are unavailable. Combined with the query results from step 1, the available chips for this node are empty.

    Note

    When the cluster scale exceeds 1,000 nodes, the ConfigMaps corresponding to cluster-info-device- and mindx-dl-switchinfo- will be sharded. Each cluster-info-device- or mindx-dl-switchinfo- contains device information for a maximum of 1,000 nodes. In this scenario, you need to perform the query operations in step 1 and step 3 on all cluster-info-device- ConfigMaps to find the detailed information of the target node, so as to confirm the available chip information of that node.

Integrating UMDK into the Container Base Image

  1. From the archive directory of the Huawei Cloud Mirror Repository, download the UMDK package in the corresponding format as needed, taking umdk-urma-*.aarch64.rpm.tar.gz as an example.

  2. Log in to the server as the root user.

  3. Create the /home/umdk directory and upload the UMDK package to this directory.

    mkdir -p /home/umdk

  4. Go to the /home/umdk directory and perform the following steps to prepare a Dockerfile.

    1. Run the following command to create a Dockerfile (example filename: "Dockerfile").

      vi Dockerfile

    2. Add the relevant instructions for UMDK installation in the file.

      • openEuler container base image, using openEuler 24.03-lts as an example.

        FROM openeuler/openeuler:24.03-lts

ARG UMDK_PKG=""
COPY ./${UMDK_PKG} /tmp

RUN yum update -y && \
    yum install -y wget unzip shadow libnl3-devel && \
    if [ -n "${UMDK_PKG}" ] && [ -f "/tmp/${UMDK_PKG}" ]; then \
        echo "installing umdk from /tmp/${UMDK_PKG}"; \
        mkdir /tmp/umdk_pkgs; \
        tar -mzxf "/tmp/${UMDK_PKG}" -C /tmp/umdk_pkgs; \
        rpm -ivh /tmp/umdk_pkgs/*.rpm; \
    else \
        echo "warning: umdk package not provided, install from yum"; \
        yum install -y umdk-urma-bin umdk-urma-devel umdk-urma-lib umdk-urma-tools; \
    fi && \
    yum clean all && \
    rm -rf /var/cache/yum && \
    rm -rf /tmp/*

      • Ubuntu container base image, using Ubuntu 24.04 as an example.

        FROM ubuntu:24.04

ARG UMDK_PKG=""

COPY ./${UMDK_PKG} /tmp/

RUN apt update && \
    apt install -y libnl-3-dev libnl-genl-3-dev && \
    if [ -n "${UMDK_PKG}" ] && [ -f "/tmp/${UMDK_PKG}" ]; then \
        echo "installing umdk from /tmp/${UMDK_PKG}"; \
        mkdir /tmp/umdk_pkgs; \
        tar -mzxf "/tmp/${UMDK_PKG}" -C /tmp/umdk_pkgs ; \
        dpkg -i /tmp/umdk_pkgs/*.deb; \
    else \
        echo "error: umdk package not provided"; \
        exit 1; \
   fi && \
   apt-get clean && \
   rm -rf /var/lib/apt/lists/* && \
   rm -rf /tmp/*

    3. After writing the new content to the Dockerfile, execute the :wq command to save the content.

  5. Go to the directory where the Dockerfile is located and execute the image build command to build the image. Note that the dot (".") at the end of the command should not be omitted.

    docker build [OPTIONS] -t image_name_system_architecture:image_tag --build-arg UMDK_PKG=UMDK package name .

    The command is explained in the following table.

    Table 1 Command parameter description

    Name Description
    OPTIONS "--no-cache" option: rebuild the image without using cache.
    -t Image name.
    image_name_system_architecture:image_tag Image name and tag. Fill in according to the actual situation.
    --build-arg UMDK_PKG UMDK package name

    For example:

    docker build --no-cache -t urma_aarch64:oe_2403lts --build-arg UMDK_PKG=umdk-urma-25.12.0-B090.sp4.aarch64.rpm.tar.gz .

    When "Successfully built xxx" appears, it indicates that the image has been built successfully.

  6. After the build is complete, run the following command to view the image information.

    docker images

    The response example is as follows:

    REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
urma_aarch64        oe_2403lts          d82746acd7f1        26 minutes ago      200MB