Star851
1340
代码
代码
Issues1539
Pull Requests2220
项目讨论
Wiki
分析
Star851
1340
已合并
optimize docker to build torch_npu #37601
SCh_zx创建于 6月4日
optimize docker to build torch_npu #37601
已合并
SCh_zx创建于 6月4日
从SCh_zx/pytorch_3v2.9.0合入到Ascend/pytorchv2.9.0
共 8 个文件变更+199-30
Mci/docker/ARM/Dockerfile+17-14
@@ -1,4 +1,4 @@
1-FROM pytorch/manylinux2_28_aarch64-builder:cpu-aarch64-2.91+FROM quay.io/pypa/manylinux_2_28_aarch64:2025.04.26-0
2 2 
3ENV PATH /usr/local/bin:$PATH3ENV PATH /usr/local/bin:$PATH
4 4 
@@ -6,12 +6,13 @@ RUN echo "alias ll='ls -l --color=auto'" >> /root/.bashrc
6 6 
7# Set pip&python7# Set pip&python
8RUN cd /usr/local/bin \8RUN cd /usr/local/bin \
9- && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3.10 \9+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3.10 \
10- && ln -sf /opt/_internal/cpython-3.11.14/bin/pip3.11 pip3.11 \10+ && ln -sf /opt/python/cp311-cp311/bin/pip3.11 pip3.11 \
11- && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3 \11+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3 \
12- && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python3.10 \12+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3.10 \
13- && ln -sf /opt/_internal/cpython-3.11.14/bin/python3.11 python3.11 \13+ && ln -sf /opt/python/cp311-cp311/bin/python3.11 python3.11 \
14- && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python314+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3 \
15+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python
15 16 
16# Set pip source17# Set pip source
17RUN mkdir /root/.pip \18RUN mkdir /root/.pip \
@@ -37,19 +38,21 @@ RUN if [ "$CONFIG_FOR_LCOV" = "1" ]; then \
37 yum install -y perl-CPAN*; \38 yum install -y perl-CPAN*; \
38 fi39 fi
39 40 
40-RUN yum install -y vim-common41+RUN yum install -y vim-common\
42+ && yum install -y ninja-build \
43+ && yum install -y dos2unix \
44+ && yum install -y gcc-toolset-14 \
45+ && echo "source /opt/rh/gcc-toolset-14/enable" >> /root/.bashrc
41 46 
42# Install pip package(build)47# Install pip package(build)
43-RUN pip3.9 install pyyaml \
44- && pip3.9 install torch==2.9.0 \
45- && pip3.9 install numpy==1.25.2
46- 
47RUN pip3.10 install pyyaml \48RUN pip3.10 install pyyaml \
48- && pip3.10 install torch==2.9.0 \49+ && pip3.10 install setuptools \
50+ && pip3.10 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
49 && pip3.10 install numpy==1.25.251 && pip3.10 install numpy==1.25.2
50 52 
51RUN pip3.11 install pyyaml \53RUN pip3.11 install pyyaml \
52- && pip3.11 install torch==2.9.0 \54+ && pip3.11 install setuptools \
55+ && pip3.11 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
53 && pip3.11 install numpy==1.25.256 && pip3.11 install numpy==1.25.2
54 57 
55WORKDIR /home58WORKDIR /home
Mci/docker/README.md+8-4
@@ -1,8 +1,8 @@
1-## Ascend Pytorch Dockerfile Repository1+# Ascend Pytorch Dockerfile Repository
2 2 
3This folder hosts the `Dockerfile` to build docker images with various platforms.3This folder hosts the `Dockerfile` to build docker images with various platforms.
4 4 
5-### Build torch_npu from Docker container5+## Build torch_npu from Docker container
6 6 
7**Clone torch-npu**7**Clone torch-npu**
8 8 
@@ -16,7 +16,9 @@ git clone https://gitcode.com/ascend/pytorch.git --depth 1
16cd pytorch/ci/docker/{arch} # {arch} for X86 or ARM16cd pytorch/ci/docker/{arch} # {arch} for X86 or ARM
17docker build -t manylinux-builder:v1 .17docker build -t manylinux-builder:v1 .
18```18```
19+ 
19If you want to configure the environment of LCOV, please build docker image like this:20If you want to configure the environment of LCOV, please build docker image like this:
21+ 
20```Shell22```Shell
21cd pytorch/ci/docker/{arch} # {arch} for X86 or ARM23cd pytorch/ci/docker/{arch} # {arch} for X86 or ARM
22docker build -t manylinux-builder:v1 --build-arg CONFIG_FOR_LCOV=1 .24docker build -t manylinux-builder:v1 --build-arg CONFIG_FOR_LCOV=1 .
@@ -28,10 +30,12 @@ docker build -t manylinux-builder:v1 --build-arg CONFIG_FOR_LCOV=1 .
28docker run -it -v /{code_path}/pytorch:/home/pytorch manylinux-builder:v1 bash30docker run -it -v /{code_path}/pytorch:/home/pytorch manylinux-builder:v1 bash
29# {code_path} is the torch_npu source code path31# {code_path} is the torch_npu source code path
30```32```
33+ 
31**Compile torch_npu**34**Compile torch_npu**
32 35 
33-Take Python 3.9 as an example36+Take Python 3.10 as an example
37+ 
34```Shell38```Shell
35cd /home/pytorch39cd /home/pytorch
36-bash ci/build.sh --python=3.940+bash ci/build.sh --python=3.10
37```41```
Mci/docker/X86/Dockerfile+15-12
@@ -1,4 +1,4 @@
1-FROM pytorch/manylinux2_28-builder:cpu-2.91+FROM quay.io/pypa/manylinux_2_28_x86_64:2025.04.26-0
2 2 
3ENV PATH /usr/local/bin:$PATH3ENV PATH /usr/local/bin:$PATH
4 4 
@@ -6,12 +6,13 @@ RUN echo "alias ll='ls -l --color=auto'" >> /root/.bashrc
6 6 
7# Set pip&python7# Set pip&python
8RUN cd /usr/local/bin \8RUN cd /usr/local/bin \
9- && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3.10 \9+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3.10 \
10- && ln -sf /opt/_internal/cpython-3.11.14/bin/pip3.11 pip3.11 \10+ && ln -sf /opt/python/cp311-cp311/bin/pip3.11 pip3.11 \
11- && ln -sf /opt/_internal/cpython-3.10.19/bin/pip3.10 pip3 \11+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3 \
12- && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python3.10 \12+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3.10 \
13- && ln -sf /opt/_internal/cpython-3.11.14/bin/python3.11 python3.11 \13+ && ln -sf /opt/python/cp311-cp311/bin/python3.11 python3.11 \
14- && ln -sf /opt/_internal/cpython-3.10.19/bin/python3.10 python314+ && ln -sf /opt/python/cp310-cp10/bin/python3.10 python3 \
15+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python
15 16 
16# Set pip source17# Set pip source
17RUN mkdir /root/.pip \18RUN mkdir /root/.pip \
@@ -37,18 +38,20 @@ RUN if [ "$CONFIG_FOR_LCOV" = "1" ]; then \
37 yum install -y perl-CPAN*; \38 yum install -y perl-CPAN*; \
38 fi39 fi
39 40 
40-RUN yum install -y vim-common41+RUN yum install -y vim-common\
42+ && yum install -y ninja-build \
43+ && yum install -y dos2unix \
44+ && yum install -y gcc-toolset-14 \
45+ && echo "source /opt/rh/gcc-toolset-14/enable" >> /root/.bashrc
41 46 
42# Install pip package(build)47# Install pip package(build)
43-RUN pip3.9 install pyyaml \
44- && pip3.9 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
45- && pip3.9 install numpy==1.25.2
46- 
47RUN pip3.10 install pyyaml \48RUN pip3.10 install pyyaml \
49+ && pip3.10 install setuptools \
48 && pip3.10 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \50 && pip3.10 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
49 && pip3.10 install numpy==1.25.251 && pip3.10 install numpy==1.25.2
50 52 
51RUN pip3.11 install pyyaml \53RUN pip3.11 install pyyaml \
54+ && pip3.11 install setuptools \
52 && pip3.11 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \55 && pip3.11 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
53 && pip3.11 install numpy==1.25.256 && pip3.11 install numpy==1.25.2
54 57 
Adocker/builder/ARM/Dockerfile+58-0
@@ -0,0 +1,58 @@
1+FROM quay.io/pypa/manylinux_2_28_aarch64:2025.04.26-0
2+ 
3+ENV PATH /usr/local/bin:$PATH
4+ 
5+RUN echo "alias ll='ls -l --color=auto'" >> /root/.bashrc
6+ 
7+# Set pip&python
8+RUN cd /usr/local/bin \
9+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3.10 \
10+ && ln -sf /opt/python/cp311-cp311/bin/pip3.11 pip3.11 \
11+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3 \
12+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3.10 \
13+ && ln -sf /opt/python/cp311-cp311/bin/python3.11 python3.11 \
14+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3 \
15+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python
16+ 
17+# Set pip source
18+RUN mkdir /root/.pip \
19+ && echo "[global]" > /root/.pip/pip.conf \
20+ && echo "index-url=https://mirrors.huaweicloud.com/repository/pypi/simple" >> /root/.pip/pip.conf \
21+ && echo "trusted-host=repo.huaweicloud.com" >> /root/.pip/pip.conf \
22+ && echo "timeout=120" >> /root/.pip/pip.conf
23+ 
24+ARG CONFIG_FOR_LCOV=0
25+ 
26+RUN if [ "$CONFIG_FOR_LCOV" = "1" ]; then \
27+ mkdir -p /etc/yum.repos.d/backup && \
28+ mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ && \
29+ curl -o /etc/yum.repos.d/Centos7-aliyun.repo https://mirrors.wlnmp.com/centos/Centos7-aliyun-altarch.repo -k && \
30+ yum clean all && \
31+ yum makecache; \
32+ fi
33+ 
34+RUN if [ "$CONFIG_FOR_LCOV" = "1" ]; then \
35+ yum install -y perl-Module-Load-Conditional && \
36+ yum install -y perl-Digest-MD5 && \
37+ yum install -y "perl(DateTime)" && \
38+ yum install -y perl-CPAN*; \
39+ fi
40+ 
41+RUN yum install -y vim-common\
42+ && yum install -y ninja-build \
43+ && yum install -y dos2unix \
44+ && yum install -y gcc-toolset-14 \
45+ && echo "source /opt/rh/gcc-toolset-14/enable" >> /root/.bashrc
46+ 
47+# Install pip package(build)
48+RUN pip3.10 install pyyaml \
49+ && pip3.10 install setuptools \
50+ && pip3.10 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
51+ && pip3.10 install numpy==1.25.2
52+ 
53+RUN pip3.11 install pyyaml \
54+ && pip3.11 install setuptools \
55+ && pip3.11 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
56+ && pip3.11 install numpy==1.25.2
57+ 
58+WORKDIR /home
Adocker/builder/README.md+41-0
@@ -0,0 +1,41 @@
1+# Ascend Pytorch Dockerfile Repository
2+ 
3+This folder hosts the `Dockerfile` to build docker images with various platforms.
4+ 
5+## Build torch_npu from Docker container
6+ 
7+**Clone torch-npu**
8+ 
9+```Shell
10+git clone https://gitcode.com/ascend/pytorch.git --depth 1
11+```
12+ 
13+**Build docker image**
14+ 
15+```Shell
16+cd pytorch/docker/builder/{arch} # {arch} for X86 or ARM
17+docker build -t manylinux-builder:v1 .
18+```
19+ 
20+If you want to configure the environment of LCOV, please build docker image like this:
21+ 
22+```Shell
23+cd pytorch/docker/builder/{arch} # {arch} for X86 or ARM
24+docker build -t manylinux-builder:v1 --build-arg CONFIG_FOR_LCOV=1 .
25+```
26+ 
27+**Enter docker Container**
28+ 
29+```Shell
30+docker run -it -v /{code_path}/pytorch:/home/pytorch manylinux-builder:v1 bash
31+# {code_path} is the torch_npu source code path
32+```
33+ 
34+**Compile torch_npu**
35+ 
36+Take Python 3.10 as an example
37+ 
38+```Shell
39+cd /home/pytorch
40+bash ci/build.sh --python=3.10
41+```
Adocker/builder/X86/Dockerfile+58-0
@@ -0,0 +1,58 @@
1+FROM quay.io/pypa/manylinux_2_28_x86_64:2025.04.26-0
2+ 
3+ENV PATH /usr/local/bin:$PATH
4+ 
5+RUN echo "alias ll='ls -l --color=auto'" >> /root/.bashrc
6+ 
7+# Set pip&python
8+RUN cd /usr/local/bin \
9+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3.10 \
10+ && ln -sf /opt/python/cp311-cp311/bin/pip3.11 pip3.11 \
11+ && ln -sf /opt/python/cp310-cp310/bin/pip3.10 pip3 \
12+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3.10 \
13+ && ln -sf /opt/python/cp311-cp311/bin/python3.11 python3.11 \
14+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python3 \
15+ && ln -sf /opt/python/cp310-cp310/bin/python3.10 python
16+ 
17+# Set pip source
18+RUN mkdir /root/.pip \
19+ && echo "[global]" > /root/.pip/pip.conf \
20+ && echo "index-url=https://mirrors.huaweicloud.com/repository/pypi/simple" >> /root/.pip/pip.conf \
21+ && echo "trusted-host=repo.huaweicloud.com" >> /root/.pip/pip.conf \
22+ && echo "timeout=120" >> /root/.pip/pip.conf
23+ 
24+ARG CONFIG_FOR_LCOV=0
25+ 
26+RUN if [ "$CONFIG_FOR_LCOV" = "1" ]; then \
27+ mkdir -p /etc/yum.repos.d/backup && \
28+ mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/backup/ && \
29+ curl -o /etc/yum.repos.d/Centos7-aliyun.repo https://mirrors.wlnmp.com/centos/Centos7-aliyun-altarch.repo -k && \
30+ yum clean all && \
31+ yum makecache; \
32+ fi
33+ 
34+RUN if [ "$CONFIG_FOR_LCOV" = "1" ]; then \
35+ yum install -y perl-Module-Load-Conditional && \
36+ yum install -y perl-Digest-MD5 && \
37+ yum install -y "perl(DateTime)" && \
38+ yum install -y perl-CPAN*; \
39+ fi
40+ 
41+RUN yum install -y vim-common\
42+ && yum install -y ninja-build \
43+ && yum install -y dos2unix \
44+ && yum install -y gcc-toolset-14 \
45+ && echo "source /opt/rh/gcc-toolset-14/enable" >> /root/.bashrc
46+ 
47+# Install pip package(build)
48+RUN pip3.10 install pyyaml \
49+ && pip3.10 install setuptools \
50+ && pip3.10 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
51+ && pip3.10 install numpy==1.25.2
52+ 
53+RUN pip3.11 install pyyaml \
54+ && pip3.11 install setuptools \
55+ && pip3.11 install torch==2.9.0+cpu --index-url https://download.pytorch.org/whl/cpu \
56+ && pip3.11 install numpy==1.25.2
57+ 
58+WORKDIR /home
Mtorch_npu/csrc/core/npu/NPUAffinityController.h+1-0
@@ -1,4 +1,5 @@
1#pragma once1#pragma once
2+#include <algorithm>
ffmh
ffmhffmh6月5日

这里include的作用是什么

likedislike
S
SCh_zx
6月5日 评论：
2#include <c10/core/Device.h>3#include <c10/core/Device.h>
3#include <torch_npu/csrc/core/npu/GetAffinityCPUInfo.h>4#include <torch_npu/csrc/core/npu/GetAffinityCPUInfo.h>
4#include <set>5#include <set>
Mtorch_npu/csrc/toolkit/profiler/common/utils.h+1-0
@@ -7,6 +7,7 @@
7#include <libgen.h>7#include <libgen.h>
8#include <fcntl.h>8#include <fcntl.h>
9#include <sys/syscall.h>9#include <sys/syscall.h>
10+#include <time.h>
10 11 
11#include <stdint.h>12#include <stdint.h>
12 13