Appendix

Edge Container Log Output Guidelines

Background

Due to the limited storage space on edge devices (such as the Atlas 500 A2 edge station), and because these devices typically use flash storage media like eMMC, which has a limited lifespan, users are advised to follow the log output recommendations for edge containers in this chapter. This will help ensure that edge containers output logs in an appropriate manner, preventing storage space from being filled too quickly—which could impact services—and avoiding premature exhaustion of the storage medium's lifespan.

Output Methods

Edge container applications running on Atlas hardware are generally managed through Kubernetes-compatible edge management platforms, such as Huawei Cloud IEF or third-party edge platforms built on KubeEdge. On such platforms, container log output methods are mainly divided into the following three types:

• Container console standard output (STDOUT and STDERR)
• (Recommended) Mounting to a host directory
• Outputting container logs directly to a log service

Container Console Standard Output

In this method, the application outputs the container log to standard output. By default, the Docker engine captures the standard output of all containers, writes it to a file in JSON format, and saves the file to the host's /var/lib/docker/containers/{containerid} directory, as shown in Figure 1.

Figure 1 Path example for the {containerid}-json.log file

(Recommended) Mounting Logs to Host Directory

In this method, the collection of edge platform logs is illustrated in Figure 2.

Figure 2 Solution architecture

Applications mount container logs to the edge host. The edge management platform provides log collection capabilities on the host and performs log rotation for host file logs.

Outputting Container Logs Directly to a Log Service

As shown in Figure 3, if a log server exists in the application environment, container logs can be directly output to the external log server, preventing logs from persisting to disk in the edge environment and minimizing the impact on hardware and other services.

Figure 3 Solution architecture

Content Mounted by Ascend Docker Runtime

Ascend Docker Runtime mounts the following directories and files into containers in read-only mode by default based on the actual environment. The mounted files must not grant write permissions for other users, nor for others in the same group.

Table 1 Default mount directories and files (Atlas 200 AI accelerator module (RC))

Table 2 Default mount directory and file (Atlas 200I SoC A1 core board)

Table 3 Default mount directories and files (Atlas 200I A2 acceleration module and Atlas 200I DK A2)

Table 4 Default mount directories and files (Atlas 500 intelligent station (model 3000))

Table 5 Default mount directories and files (Atlas 500 A2 intelligent station)

Table 6 Default mount directories and files (Atlas 350 PCIe card)

Table 7 Default mount directories and files (other devices)

Default Mount Whitelist of Ascend Docker Runtime

Ascend Docker Runtime supports custom mounting via ASCEND_RUNTIME_MOUNTS. For details, see (Optional) Configuring Custom Mounted Content. The default mount whitelists of Ascend Docker Runtime are restriced, as shown in Table 1.

Table 1 Default mount whitelists

Ascend Docker Runtime Command Description

After Ascend Docker Runtime is installed, executable tools are generated in the installation directory. The commands involved are internal commands and should not be used directly by users. The relevant commands are shown in Table 1.

Table 1 Command description

Differences Between Image and Binary Deployments

Table 1 Differences between the two installation methods

Differences Between Using ServiceAccount and KubeConfig

Table 1 Differences in Kubernetes authentication and authorization methods

Scheduling Components in High-Availability Clusters

In a production environment, Kubernetes clusters typically deploy multiple management nodes to prevent the entire cluster from becoming unavailable due to a single management node failure. Kubernetes officially provides two high-availability cluster setup solutions. For details, see the high-availability topology options in the Kubernetes documentation. The cluster scheduling components have been verified based on the official Stacked etcd topology solution. Each component can run normally in scenarios with multiple management nodes, and all functions operate correctly.

In a multi-management node scenario, you must ensure that the configurations of all management nodes are consistent, such as the cluster scheduling component images, log directories, running users, and node labels. For the installation of cluster scheduling components in a multi-management node scenario, see Installation and Deployment. For the installation of multiple management nodes, see the Kubernetes official documentation Creating Highly Available Clusters with kubeadm.

Cluster Scheduling Components Using Containerd

For Kubernetes > 1.20, using Docker as the CRI (container runtime interface) will no longer be supported. In a production environment, if a higher version of Kubernetes is required, you need to consider switching to another CRI. The cluster scheduling components have been installed and verified based on Containerd version 1.4.4, a mainstream CRI. Each component can run normally in the Containerd + Kubernetes scenario, and all functions operate correctly.

For the Containerd installation process, see the official documentation. Ascend Docker Runtime is used by default during installation. You can configure Containerd to use Ascend Docker Runtime instead of runc to automatically mount devices when starting containers. For the configuration required for Containerd, see the section on installing Ascend Docker Runtime in the Containerd scenario in Installation and Deployment.

Model Training Job Description

The table below lists constraints on training jobs based on server type when schedulers other than Volcano are used. When Volcano is used, these constraints are already satisfied during job scheduling.

Table 1 Training job instructions

Ascend Device Plugin Communication Files

Socket Files

In Ascend Device Plugin, sock files used for communication are generated. The types of sock files are as follows.

• npu.sock
• Ascend910.sock
• Ascend310.sock
• Ascend310P.sock
• davinci-mini.sock
• Ascend910-X.sock: X can be 2c, 4c, 8c, 16c, 12c.3cpu.32g, 12c.3cpu.32g.dvpp, 12c.3cpu.32g.ndvpp, 6c.1cpu.16g, 3c.0.5cpu.8g, 10c.3cpu.16g, 10c.3cpu.16g.dvpp, 10c.3cpu.16g.ndvpp, 5c.1cpu.8g, 4c.1cpu.5g, 10c.3cpu.32g, 10c.3cpu.32g.dvpp, 10c.3cpu.32g.ndvpp, and 5c.1cpu.16g
• Ascend310P-X.sock: X can be 1c, 2c, 4c, 2c.1cpu, 4c.3cpu, 4c.3cpu.ndvpp, and 4c.4cpu.dvpp

The preceding sock files are only used for communication with the local Kubernetes.

Chip Fault Code References

For detailed descriptions of chip fault codes for each product, see Table 1.

Table 1 Fault code referencs

Node Fault Code References

For detailed descriptions of node fault codes for each product, see Table 1.

Table 1 Node fault code reference documentation

Terms

This section introduces some terms in the cluster scheduling component user guide to help you better understand the document content and steps. The term descriptions are shown in Table 1.

Table 1 Terms

Public Network Address

Table 1 Description of non-public network addresses in the cluster scheduling component code

Security Statement

The code displayed for resumable training is open-source code, and the scripts involved (Python and shell) need to be set with the same user and user group. For security reasons, it is recommended that you validate input parameters, file directories, file paths, and other information.

Input parameter validation items include but are not limited to:

• Strict parameter validation and injection prevention measures are implemented for any use of external variables as part of commands.
• External variables obtained from environment variables must undergo strict validation and injection prevention measures before being used in command concatenation.
• All processes should follow the principle of least privilege to avoid severe consequences caused by injection.
• No external variables are directly used as commands in the code.
• Comply with the security specifications of various programming languages.

File path validation items include but are not limited to:

• Path length is restricted.
• There are special character filtering and anti-bypass mechanisms on paths.
• No command injection exists.
• Processes adhere to the principle of least privilege.
• No high-risk paths exist in the trustlist.
• File path authenticity is verified, and exception handling is implemented.
• Command injection is an unexpected behavior caused by controllable external variables.
• The recovery policy only supports Python 3.7 and Python 3.9.
• During script adaptation, you need to catch exceptions and handle them according to business logic.

User Information List

Update passwords periodically to avoid the risks associated with using the same password for a long time.

System Users

Users of Cluster Scheduling Component Containers

User in the nginx Container (Not Applicable in Non-security Hardening Scenarios)

Alpha Base Image Users in the Dockerfile Example

Ubuntu Base Image Users in the Dockerfile Example

ServiceAccount of Kubernetes

Table 1 List of ServiceAccounts created by components in Kubernetes

mindcluster-deploy Open-Source Repository Version Description

mindcluster-deploy is the open-source repository for the MindCluster cluster scheduling components. The repository provides MindCluster sample code and scripts for reference only and cannot be used in a production environment.

The table below describes the version compatibility.

Table 1 Version compatibility of the mindcluster-deploy repository

Kubernetes Cluster Basic Performance Tuning

The MindCluster is a functional component based on the Kubernetes ecosystem. Therefore, resumable training is supported only when the Kubernetes platform is deployed. The Kubernetes versions supported for this feature are the same as those for MindCluster cluster scheduling components, currently 1.17.x to 1.34.x (1.19.x or later is recommended).

Table 1 Configuration description

--max-request-inflight=20000
--max-mutating-requests-inflight=2000

--watch-cache=true
--watch-cache-sizes=node#1000,pod#2000,event#200,namespace#100,service#200

resources:
  requests:
    cpu: 35000m

--quota-backend-bytes=8589934592

--auto-compaction-retention

resources:
  requests:
    cpu: 20000m
    memory: 10000Mi

resources:
  requests:
    cpu: 20000m
    memory: 8Gi

kubectl edit deployment -n volcano-system volcano-scheduler

Custom Metric Development

For details, see Developer Guide - Custom Metric Development.

Preparing the Installation Environment

Notes on Installing Kubernetes

• When Kubernetes uses Calico as the cluster network plugin, the default network configuration is node-to-node mesh. In large-scale clusters, this configuration may cause excessive network load on service switches. It is recommended to configure it to reflector mode. For details, see the Calico official documentation.
• When installing Kubernetes on CentOS 7.6 and using Calico v3.24 as the cluster network plugin, the installation may fail. For related constraints, see System Requirements.
• Starting from Kubernetes 1.24, Dockershim has been removed from the Kubernetes project. If you still want to use Docker as the container engine for Kubernetes, you need to install cri-dockerd. For details, see the Docker Usage Failure When Using Kubernetes 1.24 or Later section.
• Kubernetes 1.25.10 and later versions do not support the recovery feature of virtualized NPUs.

Installing the Open-Source System

Before installing the cluster scheduling components, ensure that the following basic environment preparations are complete:

• Install Docker. Versions 18.09.x to 28.5.1 are supported. For details, see Install Docker.

• Install Containerd. Versions 1.4.x to 2.1.4 are supported. For details, see Install Containerd.

• Install Kubernetes. Kubernetes versions 1.17.x to 1.34.x are supported (version 1.19.x or later is recommended). For details, see Install Kubernetes. It is recommended to create a cluster using Kubeadm. For some issues during cluster initialization, see Kubernetes Initialization Failure. Also, you need to de-isolate the management node. The command example is as follows.

  • Kubernetes < 1.24

    • De-isolate a single node.

      kubectl taint nodes <hostname> node-role.kubernetes.io/master-
      

    • De-isolate all nodes.

      kubectl taint nodes --all node-role.kubernetes.io/master-
      

  • Kubernetes ≥ 1.24

    • De-isolate a single node.

      kubectl taint nodes <hostname> node-role.kubernetes.io/control-plane:NoSchedule-
      

    • De-isolate all nodes.

      kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-
      

  Note

  De-isolating the management node removes the taint from the master node, allowing pods to be scheduled onto the master node.