Star845
1310
代码
代码
Issues1435
Pull Requests2278
流水线
项目讨论
Wiki
分析
Star845
1310
已合并
feat(ci): add Docker image build system for builder and test environments #37457
WSs_321创建于 6月2日
feat(ci): add Docker image build system for builder and test environments #37457
已合并
WSs_321创建于 6月2日
从WSs_321/pytorchsync-pr37107-to-v2.9.0合入到Ascend/pytorchv2.9.0
共 12 个文件变更+658-7
A.ci/docker/README.md+94-0
@@ -0,0 +1,94 @@
1+# torch-npu CI Docker Images
2+ 
3+本目录管理 torch-npu 项目的 CI Docker 镜像，包括**构建镜像 (builder)** 和**测试镜像 (test)** 两类，每类分别支持 x86_64 和 aarch64 架构。
4+ 
5+## 镜像类型
6+ 
7+| 类型 | 基座 | 用途 |
8+|------|------|------|
9+| **builder (x86_64)** | `pytorch/manylinux2_28-builder:cpu-v2.9.0-rc11` | 编译构建 torch-npu wheel 包，包含完整编译工具链 |
10+| **builder (aarch64)** | `pytorch/manylinux2_28_aarch64-builder:cpu-aarch64-v2.9.0-rc11` | 编译构建 torch-npu wheel 包，包含完整编译工具链 |
11+| **test** | `ubuntu:22.04` | CI 单元测试运行环境，包含 PyTorch CPU、CANN runtime、triton-ascend 和测试框架 |
12+ 
13+## 目录结构
14+ 
15+```text
16+.ci/docker/
17+├── README.md
18+├── requirements-builder.txt # Builder 镜像 pip 依赖
19+├── requirements-test.txt # Test 镜像 pip 依赖
20+├── docker_build.sh # 构建入口脚本
21+├── common/ # 共享安装脚本
22+│ ├── install_cann.sh # 安装 CANN toolkit (支持 A1/A2/A3)
23+│ ├── install_triton.sh # 安装 triton-ascend (需传 Python 版本)
24+│ └── install_obs.sh # 安装华为 OBS util
25+├── builder/
26+│ ├── Dockerfile.x86_64
27+│ └── Dockerfile.aarch64
28+└── test/
29+ ├── Dockerfile.x86_64
30+ └── Dockerfile.aarch64
31+```
32+ 
33+## 快速构建
34+ 
35+```bash
36+# Builder 镜像 (不含 CANN)
37+./docker_build.sh torch-npu-builder-x86_64-torch2.9.0
38+./docker_build.sh torch-npu-builder-aarch64-torch2.9.0
39+ 
40+# Test 镜像 (含 CANN)
41+./docker_build.sh torch-npu-test-x86_64-cann-a1-py3.10-torch2.9.0
42+./docker_build.sh torch-npu-test-aarch64-cann-a2-py3.10-torch2.9.0
dilililiwhy
dilililiwhydilililiwhy6月27日

后续可以考虑在test镜像的tag中增加CANN的具体版本

likedislike
W
WSs_321
6月27日 评论：
43+```
44+ 
45+## Tag 命名规范
46+ 
47+参考上游 PyTorch `pytorch-linux-jammy-cuda12.4-cudnn9-py3-gcc11` 模式，tag 即为最终镜像名：
48+ 
49+**Builder**（不含 CANN）：
50+ 
51+```text
52+torch-npu-builder-<ARCH>-torch<PYTORCH_VERSION>
53+```
54+ 
55+```bash
56+./docker_build.sh torch-npu-builder-x86_64-torch2.9.0
57+# ^ ^ ^ ^
58+# | | | └── PyTorch 版本 (torch2.9.0)
59+# | | └── 架构
60+# | └── 镜像类型
61+# └── 固定前缀
62+```
63+ 
64+**Test**（含 CANN runtime）：
65+ 
66+```text
67+torch-npu-test-<ARCH>-cann<CHIP>-py<PYTHON_VERSION>-torch<PYTORCH_VERSION>
68+```
69+ 
70+```bash
71+./docker_build.sh torch-npu-test-x86_64-cann-a1-py3.10-torch2.9.0
72+# ^ ^ ^ ^ ^
73+# | | | | └── PyTorch 版本 (torch2.9.0)
74+# | | | └── Python 版本 (py3.10)
75+# | | └── CANN 芯片 (A1/A2/A3)
76+# | └── 架构
77+# └── 镜像类型
78+```
79+ 
80+| 字段 | 可选值 |
81+|------|--------|
82+| IMAGE_TYPE | builder, test |
83+| ARCH | x86_64, aarch64 |
84+| CHIP | A1 (Ascend 910), A2 (Ascend 910b), A3 (仅 test) |
85+| PYTHON_VERSION | 3.10 (仅 test) |
86+| PYTORCH_VERSION | 2.9.0 |
87+ 
88+## CANN 芯片映射
89+ 
90+| CANN_CHIP | 芯片 | CANN 版本 |
91+|-----------|------|----------|
92+| A1 | Ascend 910 | 9.1.0-beta.3 |
93+| A2 | Ascend 910b | 9.1.0-beta.3 |
94+| A3 | Ascend A3 | 9.1.0-beta.3 |
A.ci/docker/builder/Dockerfile.aarch64+60-0
@@ -0,0 +1,60 @@
1+FROM pytorch/manylinux2_28_aarch64-builder:cpu-aarch64-v2.9.0-rc11
2+ 
3+ARG PYTORCH_VERSION=2.9.0
4+ 
5+ENV PATH=/usr/local/bin:$PATH
6+ENV AUDITWHEEL_PLAT=manylinux_2_28_aarch64
7+ENV PYTORCH_VERSION=${PYTORCH_VERSION}
8+ 
9+COPY requirements-builder.txt /opt/buildtools/
10+ 
11+# Set pip & python symlinks
12+RUN cd /usr/local/bin \
13+ && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3.10 \
14+ && ln -sf /opt/_internal/cpython-3.11.14/bin/pip3.11 pip3.11 \
15+ && ln -sf /opt/_internal/cpython-3.12.12/bin/pip3.12 pip3.12 \
16+ && ln -sf /opt/_internal/cpython-3.13.8/bin/pip3.13 pip3.13 \
17+ && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3 \
18+ && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python3.10 \
19+ && ln -sf /opt/_internal/cpython-3.11.14/bin/python3.11 python3.11 \
20+ && ln -sf /opt/_internal/cpython-3.12.12/bin/python3.12 python3.12 \
21+ && ln -sf /opt/_internal/cpython-3.13.8/bin/python3.13 python3.13 \
22+ && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python3
23+ 
24+# Install PyTorch from official source, then build requirements from PyPI for each python version
25+RUN pip3.10 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION} \
26+ && pip3.10 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
27+ && pip3.11 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION} \
28+ && pip3.11 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
29+ && pip3.12 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION} \
30+ && pip3.12 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
31+ && pip3.13 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION} \
32+ && pip3.13 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
33+ && pip3.13 install --no-cache-dir auditwheel==5.4.0 \
34+ && ln -sf /opt/_internal/cpython-3.13.8/bin/auditwheel /usr/local/bin/auditwheel
35+ 
36+# Install system build tools
37+RUN echo "alias ll='ls -l --color=auto'" >> /root/.bashrc \
38+ && yum install -y ninja-build binutils lld mold dos2unix gcc gcc-c++ make cmake3 wget tar unzip elfutils java-1.8.0-openjdk-devel \
39+ && cd /tmp \
40+ && wget https://github.com/ccache/ccache/releases/download/v4.10/ccache-4.10.tar.gz \
41+ && tar -xzf ccache-4.10.tar.gz \
42+ && cd ccache-4.10 \
43+ && mkdir build \
44+ && cd build \
45+ && cmake3 .. \
46+ && make -j$(nproc) \
47+ && make install \
48+ && cd /tmp \
49+ && rm -rf ccache-4.10* \
50+ && ccache --version \
51+ && yum update -y \
52+ && yum clean all
53+ 
54+# Set timezone
55+RUN rm -f /etc/localtime \
56+ && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
57+ && echo 'Asia/Shanghai' >/etc/timezone \
58+ && echo "export TZ='Asia/Shanghai'" >>/etc/profile
59+ 
60+WORKDIR /home
A.ci/docker/builder/Dockerfile.x86_64+72-0
@@ -0,0 +1,72 @@
1+FROM pytorch/manylinux2_28-builder:cpu-v2.9.0-rc11
2+ 
3+ARG PYTORCH_VERSION=2.9.0
4+ 
5+ENV PATH=/usr/local/bin:$PATH
6+ENV AUDITWHEEL_PLAT=manylinux_2_28_x86_64
7+ENV PYTORCH_VERSION=${PYTORCH_VERSION}
8+ 
9+COPY requirements-builder.txt /opt/buildtools/
10+ 
11+# Set pip & python symlinks
12+RUN cd /usr/local/bin \
13+ && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3.10 \
14+ && ln -sf /opt/_internal/cpython-3.11.14/bin/pip3.11 pip3.11 \
15+ && ln -sf /opt/_internal/cpython-3.12.12/bin/pip3.12 pip3.12 \
16+ && ln -sf /opt/_internal/cpython-3.13.8/bin/pip3.13 pip3.13 \
17+ && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3 \
18+ && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python3.10 \
19+ && ln -sf /opt/_internal/cpython-3.11.14/bin/python3.11 python3.11 \
20+ && ln -sf /opt/_internal/cpython-3.12.12/bin/python3.12 python3.12 \
21+ && ln -sf /opt/_internal/cpython-3.13.8/bin/python3.13 python3.13 \
22+ && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python3
23+ 
24+# Install PyTorch from official source, then build requirements from PyPI for each python version
25+RUN pip3.10 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION}+cpu \
26+ && pip3.10 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
27+ && pip3.11 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION}+cpu \
28+ && pip3.11 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
29+ && pip3.12 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION}+cpu \
30+ && pip3.12 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
31+ && pip3.13 install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION}+cpu \
32+ && pip3.13 install --no-cache-dir -r /opt/buildtools/requirements-builder.txt \
33+ && pip3.13 install --no-cache-dir auditwheel==5.4.0 \
34+ && ln -sf /opt/_internal/cpython-3.13.8/bin/auditwheel /usr/local/bin/auditwheel
35+ 
36+# Install system build tools
37+RUN yum remove -y ius-release epel-release 2>/dev/null || true \
38+ && rm -rf /etc/yum.repos.d/ius*.repo /etc/yum.repos.d/epel*.repo \
39+ && yum clean all && rm -rf /var/cache/dnf /var/cache/yum \
40+ && echo "alias ll='ls -l --color=auto'" >> /root/.bashrc \
41+ && yum install -y binutils lld dos2unix gcc gcc-c++ make cmake3 wget tar unzip elfutils java-1.8.0-openjdk-devel \
42+ && cd /tmp \
43+ && wget -q https://github.com/ninja-build/ninja/releases/download/v1.12.1/ninja-linux.zip \
44+ && unzip ninja-linux.zip \
45+ && cp ninja /usr/local/bin/ && chmod +x /usr/local/bin/ninja \
46+ && cd /tmp \
47+ && wget -q https://github.com/rui314/mold/archive/refs/tags/v2.32.1.tar.gz \
48+ && tar -xf v2.32.1.tar.gz \
49+ && cd mold-2.32.1 \
50+ && cmake -DCMAKE_BUILD_TYPE=Release -DMOLD_MOSTLY_STATIC=ON . \
51+ && make -j$(nproc) && make install \
52+ && cd /tmp \
53+ && wget https://github.com/ccache/ccache/releases/download/v4.10/ccache-4.10.tar.gz \
54+ && tar -xzf ccache-4.10.tar.gz \
55+ && cd ccache-4.10 \
56+ && mkdir build \
57+ && cd build \
58+ && cmake3 .. \
59+ && make -j$(nproc) \
60+ && make install \
61+ && cd /tmp && rm -rf /tmp/* \
62+ && ninja --version && mold --version && ccache --version \
63+ && yum clean all \
64+ && yum update -y
65+ 
66+# Set timezone
67+RUN rm -f /etc/localtime \
68+ && cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
69+ && echo 'Asia/Shanghai' >/etc/timezone \
70+ && echo "export TZ='Asia/Shanghai'" >>/etc/profile
71+ 
72+WORKDIR /home
A.ci/docker/common/install_cann.sh+109-0
@@ -0,0 +1,109 @@
1+#!/usr/bin/bash
2+# Install CANN toolkit for Ascend NPU.
3+# Usage: CANN_CHIP=A1 ./install_cann.sh
4+# CANN_CHIP: A1 (Ascend 910), A2 (Ascend 910b), A3 (Ascend A3)
5+# Automatically detects architecture (x86_64 / aarch64).
6+ 
7+set -e
8+ 
9+CANN_CHIP="${CANN_CHIP:-A1}"
10+ARCH=$(uname -m)
11+ 
12+CANN_BASE_URL="https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%209.1.T6"
13+CANN_VERSION="9.1.0-beta.3"
14+ 
15+case "${ARCH}_${CANN_CHIP}" in
16+ # x86_64
17+ x86_64_A1)
18+ TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_${CANN_VERSION}_linux-x86_64.run"
19+ OPS_URL="${CANN_BASE_URL}/Ascend-cann-910-ops_${CANN_VERSION}_linux-x86_64.run"
20+ NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_${CANN_VERSION}_linux-x86_64.run"
21+ OPS_GLOB="Ascend-cann-910*"
22+ SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
23+ ;;
24+ x86_64_A2)
25+ TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_${CANN_VERSION}_linux-x86_64.run"
26+ OPS_URL="${CANN_BASE_URL}/Ascend-cann-910b-ops_${CANN_VERSION}_linux-x86_64.run"
27+ NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_${CANN_VERSION}_linux-x86_64.run"
28+ OPS_GLOB="Ascend-cann-910b-ops*"
29+ SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
30+ ;;
31+ x86_64_A3)
32+ TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_${CANN_VERSION}_linux-x86_64.run"
33+ OPS_URL="${CANN_BASE_URL}/Ascend-cann-A3-ops_${CANN_VERSION}_linux-x86_64.run"
34+ NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_${CANN_VERSION}_linux-x86_64.run"
35+ OPS_GLOB="Ascend-cann-A3-ops*"
36+ SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
37+ ;;
38+ # aarch64
39+ aarch64_A1)
40+ TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_${CANN_VERSION}_linux-aarch64.run"
41+ OPS_URL="${CANN_BASE_URL}/Ascend-cann-910-ops_${CANN_VERSION}_linux-aarch64.run"
42+ NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_${CANN_VERSION}_linux-aarch64.run"
43+ OPS_GLOB="Ascend-cann-910*"
44+ SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
45+ ;;
46+ aarch64_A2)
47+ TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_${CANN_VERSION}_linux-aarch64.run"
48+ OPS_URL="${CANN_BASE_URL}/Ascend-cann-910b-ops_${CANN_VERSION}_linux-aarch64.run"
49+ NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_${CANN_VERSION}_linux-aarch64.run"
50+ OPS_GLOB="Ascend-cann-910b-ops*"
51+ SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
52+ ;;
53+ aarch64_A3)
54+ TOOLKIT_URL="${CANN_BASE_URL}/Ascend-cann-toolkit_${CANN_VERSION}_linux-aarch64.run"
55+ OPS_URL="${CANN_BASE_URL}/Ascend-cann-A3-ops_${CANN_VERSION}_linux-aarch64.run"
56+ NNAL_URL="${CANN_BASE_URL}/Ascend-cann-nnal_${CANN_VERSION}_linux-aarch64.run"
57+ OPS_GLOB="Ascend-cann-A3-ops*"
58+ SET_ENV_PATH="/usr/local/Ascend/cann/set_env.sh"
59+ ;;
60+ *)
61+ echo "Unsupported combination: ${ARCH} + ${CANN_CHIP}"
62+ exit 1
63+ ;;
64+esac
65+ 
66+echo "Installing CANN ${CANN_CHIP} for ${ARCH}..."
67+ 
68+echo "=== Creating HwHiAiUser user and group ==="
69+groupadd -f HwHiAiUser
70+id -u HwHiAiUser >/dev/null 2>&1 || useradd -g HwHiAiUser -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash
71+ 
72+rm -rf cann
73+mkdir -p cann && cd cann
74+ 
75+echo "=== Downloading CANN packages ==="
76+curl -O "${TOOLKIT_URL}"
77+curl -O "${OPS_URL}"
78+curl -O "${NNAL_URL}"
79+echo "Download complete."
80+ 
81+chmod +x Ascend-cann*.run
82+ 
83+echo "=== Installing CANN toolkit ==="
84+./Ascend-cann-toolkit*.run --full --quiet --install-path=/usr/local/Ascend
85+source "${SET_ENV_PATH}"
86+echo "toolkit install success"
87+ 
88+echo "=== Installing CANN ops ==="
89+./${OPS_GLOB}.run --install --quiet --install-path=/usr/local/Ascend
90+echo "ops install success"
91+ 
92+echo "=== Installing CANN nnal ==="
93+./Ascend-cann-nnal*.run --install --quiet --install-path=/usr/local/Ascend
94+source /usr/local/Ascend/nnal/atb/set_env.sh
95+echo "nnal install success"
96+ 
97+# Some CANN versions install to versioned paths (e.g. cann-9.0.0-beta.2)
98+# instead of /usr/local/Ascend/cann/. Fix broken symlinks so runtime
99+# sourcing of set_env.sh works.
100+if [ ! -f /usr/local/Ascend/cann/set_env.sh ]; then
101+ CANN_REAL_DIR=$(ls -d /usr/local/Ascend/cann-* 2>/dev/null | head -1)
102+ if [ -n "${CANN_REAL_DIR}" ]; then
103+ ln -sf "${CANN_REAL_DIR}" /usr/local/Ascend/cann
104+ echo "Fixed: linked ${CANN_REAL_DIR} -> /usr/local/Ascend/cann"
105+ fi
106+fi
107+ 
108+rm -rf *
109+echo "CANN ${CANN_CHIP} installation complete."
A.ci/docker/common/install_obs.sh+21-0
@@ -0,0 +1,21 @@
1+#!/usr/bin/bash
2+# Install Huawei OBS util for object storage access.
3+ 
4+set -e
5+ 
6+ARCH=$(uname -m)
7+case "${ARCH}" in
8+ x86_64) OBS_ARCH="amd64" ;;
9+ aarch64) OBS_ARCH="arm64" ;;
10+ *) echo "Unsupported architecture: ${ARCH}"; exit 1 ;;
11+esac
12+ 
13+OBS_URL="https://obs-community.obs.cn-north-1.myhuaweicloud.com/obsutil/current/obsutil_linux_${OBS_ARCH}.tar.gz"
14+ 
15+wget -q "${OBS_URL}"
16+mkdir -p /usr/local/obsutil
17+tar -zxf "obsutil_linux_${OBS_ARCH}.tar.gz" -C /usr/local/obsutil/
18+rm -f "obsutil_linux_${OBS_ARCH}.tar.gz"
19+ln -sf /usr/local/obsutil/obsutil_linux_${OBS_ARCH}_*/obsutil /usr/local/bin/obsutil
20+ 
21+echo "OBS util installed."
A.ci/docker/common/install_triton.sh+19-0
@@ -0,0 +1,19 @@
1+#!/usr/bin/bash
2+# Install triton-ascend for NPU.
3+# Usage: ./install_triton.sh <PYTHON_VERSION>
4+# PYTHON_VERSION: e.g. 3.10, 3.11, 3.12, 3.13
5+ 
6+set -e
7+ 
8+TRITON_VERSION="${TRITON_VERSION:-3.2.1}"
9+PYTHON_VERSION="${1:?Usage: $0 <PYTHON_VERSION> (e.g. 3.10)}"
10+ 
11+ARCH=$(uname -m)
12+PY_SHORT=$(echo "${PYTHON_VERSION}" | tr -d '.')
13+ 
14+TRITON_WHL="triton_ascend-${TRITON_VERSION}-cp${PY_SHORT}-cp${PY_SHORT}-manylinux_2_27_${ARCH}.manylinux_2_28_${ARCH}.whl"
15+TRITON_URL="https://gitcode.com/Ascend/triton-ascend/releases/download/v${TRITON_VERSION}/${TRITON_WHL}"
16+ 
17+echo "Installing triton-ascend ${TRITON_VERSION} for Python ${PYTHON_VERSION} (${ARCH})..."
18+pip3 install --no-cache-dir "${TRITON_URL}"
19+echo "triton-ascend installed."
A.ci/docker/docker_build.sh+115-0
@@ -0,0 +1,115 @@
1+#!/usr/bin/bash
2+# Build torch-npu CI Docker images.
3+#
4+# Usage:
5+# ./docker_build.sh <TAG>
6+#
7+# Builder: torch-npu-builder-<ARCH>-torch<PYTORCH_VERSION>
8+# Test: torch-npu-test-<ARCH>-cann<CHIP>-py<PYTHON_VERSION>-torch<PYTORCH_VERSION>
9+#
10+# Examples:
11+# ./docker_build.sh torch-npu-builder-x86_64-torch2.9.0
12+# ./docker_build.sh torch-npu-test-aarch64-cann-a2-py3.10-torch2.9.0
13+#
14+# Reference: pytorch/pytorch .ci/docker/build.sh
15+ 
16+set -ex
17+ 
18+tag="${1:?Usage: $0 <TAG>}"
19+shift
20+ 
21+case "$tag" in
22+ torch-npu-builder-x86_64-torch2.9.0)
23+ IMAGE_TYPE=builder
24+ ARCH=x86_64
25+ PYTORCH_VERSION=2.9.0
26+ ;;
27+ torch-npu-builder-aarch64-torch2.9.0)
28+ IMAGE_TYPE=builder
29+ ARCH=aarch64
30+ PYTORCH_VERSION=2.9.0
31+ ;;
32+ torch-npu-test-x86_64-cann-a1-py3.10-torch2.9.0)
33+ IMAGE_TYPE=test
34+ ARCH=x86_64
35+ CANN_CHIP=A1
36+ PYTHON_VERSION=3.10
37+ PYTORCH_VERSION=2.9.0
38+ ;;
39+ torch-npu-test-x86_64-cann-a2-py3.10-torch2.9.0)
40+ IMAGE_TYPE=test
41+ ARCH=x86_64
42+ CANN_CHIP=A2
43+ PYTHON_VERSION=3.10
44+ PYTORCH_VERSION=2.9.0
45+ ;;
46+ torch-npu-test-x86_64-cann-a3-py3.10-torch2.9.0)
47+ IMAGE_TYPE=test
48+ ARCH=x86_64
49+ CANN_CHIP=A3
50+ PYTHON_VERSION=3.10
51+ PYTORCH_VERSION=2.9.0
52+ ;;
53+ torch-npu-test-aarch64-cann-a1-py3.10-torch2.9.0)
54+ IMAGE_TYPE=test
55+ ARCH=aarch64
56+ CANN_CHIP=A1
57+ PYTHON_VERSION=3.10
58+ PYTORCH_VERSION=2.9.0
59+ ;;
60+ torch-npu-test-aarch64-cann-a2-py3.10-torch2.9.0)
61+ IMAGE_TYPE=test
62+ ARCH=aarch64
63+ CANN_CHIP=A2
64+ PYTHON_VERSION=3.10
65+ PYTORCH_VERSION=2.9.0
66+ ;;
67+ torch-npu-test-aarch64-cann-a3-py3.10-torch2.9.0)
68+ IMAGE_TYPE=test
69+ ARCH=aarch64
70+ CANN_CHIP=A3
71+ PYTHON_VERSION=3.10
72+ PYTORCH_VERSION=2.9.0
73+ ;;
74+ *)
75+ echo "Unknown tag: ${tag}"
76+ echo " Builder: torch-npu-builder-<x86_64|aarch64>-torch2.9.0"
77+ echo " Test: torch-npu-test-<x86_64|aarch64>-cann<A1|A2|A3>-py3.10-torch2.9.0"
78+ exit 1
79+ ;;
80+esac
81+ 
82+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
83+DOCKERFILE="${SCRIPT_DIR}/${IMAGE_TYPE}/Dockerfile.${ARCH}"
84+ 
85+if [[ ! -f "${DOCKERFILE}" ]]; then
86+ echo "Dockerfile not found: ${DOCKERFILE}"
87+ exit 1
88+fi
89+ 
90+BUILD_ARGS=(
91+ --build-arg PYTORCH_VERSION="${PYTORCH_VERSION}"
92+)
93+if [[ -n "${CANN_CHIP:-}" ]]; then
94+ BUILD_ARGS+=(--build-arg CANN_CHIP="${CANN_CHIP}")
95+fi
96+if [[ -n "${PYTHON_VERSION:-}" ]]; then
97+ BUILD_ARGS+=(--build-arg PYTHON_VERSION="${PYTHON_VERSION}")
98+fi
99+ 
100+TIMESTAMP="${TIMESTAMP:-$(TZ=Asia/Shanghai date +%Y%m%d%H%M)}"
101+IMAGE_TAG="${tag}-${TIMESTAMP}"
102+ 
103+echo "Building ${IMAGE_TAG} ..."
104+echo " Dockerfile: ${DOCKERFILE}"
105+echo " PyTorch: ${PYTORCH_VERSION}"
106+[[ -n "${PYTHON_VERSION:-}" ]] && echo " Python: ${PYTHON_VERSION}"
107+[[ -n "${CANN_CHIP:-}" ]] && echo " CANN chip: ${CANN_CHIP}"
108+ 
109+docker build \
110+ -f "${DOCKERFILE}" \
111+ -t "${IMAGE_TAG}" \
112+ "${BUILD_ARGS[@]}" \
113+ "${SCRIPT_DIR}"
114+ 
115+echo "Image built: ${IMAGE_TAG}"
A.ci/docker/requirements-builder.txt+5-0
@@ -0,0 +1,5 @@
1+numpy==1.26.4
2+pybind11==3.0.1
3+pyyaml==6.0.3
4+setuptools==75.3.2
5+wheel==0.45.1
D.ci/docker/requirements-ci.txt+0-7
@@ -1,7 +0,0 @@
1-# Python dependencies required for unit tests
2- 
3-mypy==1.9.0
4-# Pin MyPy version because new errors are likely to appear with each release
5-#Description: linter
6-#Pinned versions: 1.9.0
7-#test that import: test_typing.py, test_type_hints.py
A.ci/docker/requirements-test.txt+38-0
@@ -0,0 +1,38 @@
1+--index-url https://download.pytorch.org/whl/cpu
2+--extra-index-url https://pypi.org/simple
3+-f https://data.pyg.org/whl/torch-2.9.0+cpu.html
4+ 
5+coverage==7.14.0
6+hypothesis==6.152.7
7+beartype==0.17.0
8+expecttest==0.1.3
9+numpy==1.26.4
10+ 
11+mypy==1.16.0
12+onnx==1.18.0
13+onnxruntime==1.22.1
14+onnxscript==0.4.0
15+onnx-ir==0.2.1
16+ 
17+Pillow==10.3.0
18+requests==2.32.0
19+torch_geometric==2.5.3
20+torch-scatter==2.1.2
21+transformers==4.40.0
22+pytest==8.3.2
23+parameterized==0.9.0
24+ml-dtypes==0.5.1
25+protobuf==5.29.4
26+ 
27+torchvision==0.24.0
28+ 
29+pybind11==3.0.1
30+tabulate==0.9.0
31+z3-solver==4.13.0.0
32+psutil==6.0.0
33+packaging==25.0
34+decorator==5.1.1
35+scipy==1.13.1
36+unittest-xml-reporting==4.0.0
37+importlib_metadata==9.0.0
38+tensorboard==2.16.2
A.ci/docker/test/Dockerfile.aarch64+63-0
@@ -0,0 +1,63 @@
1+FROM ubuntu:22.04
2+ 
3+ARG PYTORCH_VERSION=2.9.0
4+ARG CANN_CHIP=A2
5+ARG PYTHON_VERSION=3.10
6+ 
7+ENV ETCD_UNSUPPORTED_ARCH=arm64
8+ENV DEBIAN_FRONTEND=noninteractive
9+ENV TZ=Asia/Shanghai
10+ENV PATH=/usr/local/bin:$PATH
11+ENV PYTORCH_VERSION=${PYTORCH_VERSION}
12+ENV CANN_CHIP=${CANN_CHIP}
13+ 
14+COPY common/ /opt/buildtools/
15+COPY requirements-test.txt /opt/buildtools/
16+ 
17+RUN apt-get update \
18+ && apt-get install -y --no-install-recommends \
19+ build-essential \
20+ ca-certificates \
21+ curl \
22+ dos2unix \
23+ gcc \
24+ g++ \
25+ git \
26+ make \
27+ python3 \
28+ python3-dev \
29+ python3-pip \
30+ python3-venv \
31+ tar \
32+ tzdata \
33+ unzip \
34+ vim \
35+ wget \
36+ openjdk-8-jdk-headless \
37+ && wget https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-arm64.tar.gz \
38+ && tar -zxf etcd-v3.4.3-linux-arm64.tar.gz \
39+ && mv etcd-v3.4.3-linux-arm64/etcd /usr/local/bin/ \
40+ && python3 -m pip install python-etcd \
41+ && etcd --version \
42+ && apt-get upgrade -y \
43+ && apt-get clean \
44+ && ln -sf /usr/bin/python3 /usr/bin/python \
45+ && rm -rf /var/lib/apt/lists/*
46+ 
47+# Upgrade pip/setuptools/wheel
48+RUN python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
49+ 
50+# Install CANN and OBS
51+RUN chmod -R 755 /opt/buildtools/* \
52+ && dos2unix /opt/buildtools/* \
53+ && /opt/buildtools/install_cann.sh \
54+ && /opt/buildtools/install_obs.sh
55+ 
56+# Install triton-ascend
57+RUN /opt/buildtools/install_triton.sh 3.10
58+ 
59+# Install PyTorch from official source, then test requirements from PyPI
60+RUN python3 -m pip install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION} \
61+ && python3 -m pip install --no-cache-dir --extra-index-url https://download.pytorch.org/whl/cpu -r /opt/buildtools/requirements-test.txt
62+ 
63+WORKDIR /home
A.ci/docker/test/Dockerfile.x86_64+62-0
@@ -0,0 +1,62 @@
1+FROM ubuntu:22.04
2+ 
3+ARG PYTORCH_VERSION=2.9.0
4+ARG CANN_CHIP=A1
5+ARG PYTHON_VERSION=3.10
6+ 
7+ENV DEBIAN_FRONTEND=noninteractive
8+ENV TZ=Asia/Shanghai
9+ENV PATH=/usr/local/bin:$PATH
10+ENV PYTORCH_VERSION=${PYTORCH_VERSION}
11+ENV CANN_CHIP=${CANN_CHIP}
12+ 
13+COPY common/ /opt/buildtools/
14+COPY requirements-test.txt /opt/buildtools/
15+ 
16+RUN apt-get update \
17+ && apt-get install -y --no-install-recommends \
18+ build-essential \
19+ ca-certificates \
20+ curl \
21+ dos2unix \
22+ gcc \
23+ g++ \
24+ git \
25+ make \
26+ python3 \
27+ python3-dev \
28+ python3-pip \
29+ python3-venv \
30+ tar \
31+ tzdata \
32+ unzip \
33+ vim \
34+ wget \
35+ openjdk-8-jdk-headless \
36+ && wget https://github.com/etcd-io/etcd/releases/download/v3.4.3/etcd-v3.4.3-linux-amd64.tar.gz \
37+ && tar -zxf etcd-v3.4.3-linux-amd64.tar.gz \
38+ && mv etcd-v3.4.3-linux-amd64/etcd /usr/local/bin/ \
39+ && python3 -m pip install python-etcd \
40+ && etcd --version \
41+ && apt-get upgrade -y \
42+ && apt-get clean \
43+ && ln -sf /usr/bin/python3 /usr/bin/python \
44+ && rm -rf /var/lib/apt/lists/*
45+ 
46+# Upgrade pip/setuptools/wheel
47+RUN python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
48+ 
49+# Install CANN and OBS
50+RUN chmod -R 755 /opt/buildtools/* \
51+ && dos2unix /opt/buildtools/* \
52+ && /opt/buildtools/install_cann.sh \
53+ && /opt/buildtools/install_obs.sh
54+ 
55+# Install triton-ascend
56+RUN /opt/buildtools/install_triton.sh 3.10
57+ 
58+# Install PyTorch from official source, then test requirements from PyPI
59+RUN python3 -m pip install --no-cache-dir --index-url https://download.pytorch.org/whl/cpu torch==${PYTORCH_VERSION} \
60+ && python3 -m pip install --no-cache-dir --extra-index-url https://download.pytorch.org/whl/cpu -r /opt/buildtools/requirements-test.txt
61+ 
62+WORKDIR /home