已合并
fix: 闭源包按分支拉取，支持 --bundle_branch 与自动探测(#169) #452
sinat_31531339创建于 7月25日
fix: 闭源包按分支拉取，支持 --bundle_branch 与自动探测(#169) #452
已合并
共 8 个文件变更+762-22
| @@ -124,8 +124,9 @@ bash build.sh --cann_3rd_lib_path=${third_party_path} | |||
| 124 | 124 | ||
| 125 | - `--cann_3rd_lib_path`：第三方库存储目录，默认值为 `./third_party`。若本地不存在第三方库，编译脚本将自动从 gitcode 开源仓库下载各第三方库源码。 | 125 | - `--cann_3rd_lib_path`：第三方库存储目录，默认值为 `./third_party`。若本地不存在第三方库，编译脚本将自动从 gitcode 开源仓库下载各第三方库源码。 |
| 126 | - 编译过程中会自动下载闭源二进制包，该包含有保证功能正常运行所需的库及头文件，且仅提供 release 版本，**即使编译选项指定为 debug，也只会下载 release 版本的 tar 包**。 | 126 | - 编译过程中会自动下载闭源二进制包，该包含有保证功能正常运行所需的库及头文件，且仅提供 release 版本，**即使编译选项指定为 debug，也只会下载 release 版本的 tar 包**。 |
| 127 | +- 闭源二进制包按分支拉取：不指定时，编译脚本会依据当前 git 提交自动探测所属发布分支（从 `master` 拉出的分支拉 master 包，从 9.1.0 线拉出的分支拉 9.1.0 包），探测不出时回退 `master`。也可通过 `--bundle_branch=<NAME>` 显式指定分支，个人分支探测不准时建议显式指定。当前 OBS 上提供包的分支为 `master` 与 `9.1.0`；指定其它分支会在配置阶段报错。 | ||
| 127 | - 编译过程中会通过 `git clone` 拉取 `msprof` 和 `msprobe` 子仓（分别用于构建 msprof 分析 wheel 和同步 msaccucmp 工具）。子仓源码位于 gitcode，使用 HTTPS 协议克隆前需[配置 gitcode 个人访问令牌](https://gitcode.com/setting/token-classic)以替代登录密码，否则克隆会失败。 | 128 | - 编译过程中会通过 `git clone` 拉取 `msprof` 和 `msprobe` 子仓（分别用于构建 msprof 分析 wheel 和同步 msaccucmp 工具）。子仓源码位于 gitcode，使用 HTTPS 协议克隆前需[配置 gitcode 个人访问令牌](https://gitcode.com/setting/token-classic)以替代登录密码，否则克隆会失败。 |
| 128 | -- 若编译环境无法访问网络，请参考[离线编译环境准备](./docs/zh/quick_install.md#离线编译环境准备)提前完成依赖包的下载与配置，并通过 `--cann_3rd_lib_path` 参数指定依赖包所在目录后再执行编译。 | 129 | +- 若编译环境无法访问网络，请参考[离线编译环境准备](./docs/zh/quick_install.md#离线编译环境准备)提前完成依赖包的下载与配置，并通过 `--cann_3rd_lib_path` 参数指定依赖包所在目录后再执行编译。离线预置脚本 `cmake/download_libs.py` 同样支持 `--bundle_branch` 指定要预置的闭源包分支（默认自动探测），须与联编时的分支保持一致。 |
| 129 | - 闭源二进制包会解压到仓库根目录的 `bundle/` 下。若 `bundle/` 已存在且非空，构建会复用该目录并跳过下载；如需强制重新下载或修复残缺的 `bundle/` 目录，可执行 `bash build.sh --make_clean` 后重新编译，也可手动删除 `bundle/` 后再次执行 `bash build.sh`。 | 130 | - 闭源二进制包会解压到仓库根目录的 `bundle/` 下。若 `bundle/` 已存在且非空，构建会复用该目录并跳过下载；如需强制重新下载或修复残缺的 `bundle/` 目录，可执行 `bash build.sh --make_clean` 后重新编译，也可手动删除 `bundle/` 后再次执行 `bash build.sh`。 |
| 130 | - 更多编译参数请通过 `bash build.sh -h` 查看。 | 131 | - 更多编译参数请通过 `bash build.sh -h` 查看。 |
| 131 | 132 | ||
| @@ -125,8 +125,9 @@ bash build.sh --cann_3rd_lib_path=${third_party_path} | |||
| 125 | Parameters: | 125 | Parameters: |
| 126 | - `--cann_3rd_lib_path`: The directory for storing third-party libraries. The default value is `./third_party`. If third-party libraries do not exist locally, the build script automatically downloads the source code of each third-party library from the gitcode open source repository. | 126 | - `--cann_3rd_lib_path`: The directory for storing third-party libraries. The default value is `./third_party`. If third-party libraries do not exist locally, the build script automatically downloads the source code of each third-party library from the gitcode open source repository. |
| 127 | - The build process automatically downloads closed-source binary packages that contain the libraries and header files required for normal operation. Only release versions are provided. **Even if the build option specifies debug, only the release version tar package is downloaded**. | 127 | - The build process automatically downloads closed-source binary packages that contain the libraries and header files required for normal operation. Only release versions are provided. **Even if the build option specifies debug, only the release version tar package is downloaded**. |
| 128 | +- Closed-source binary packages are fetched per branch. When not specified, the build script detects the release branch the current git commit belongs to (branches cut from `master` fetch the master package; branches on the 9.1.0 line fetch the 9.1.0 package), falling back to `master` when detection fails. You can also specify the branch explicitly with `--bundle_branch=<NAME>`, which is recommended when detection is inaccurate for personal branches. Branches with packages currently published on OBS are `master` and `9.1.0`; specifying any other branch fails at the configuration stage. | ||
| 128 | - The build process clones the `msprof` and `msprobe` submodules via `git clone` (used for building the msprof analysis wheel and syncing the msaccucmp tool, respectively). These submodules are hosted on gitcode and require a [gitcode personal access token](https://gitcode.com/setting/token-classic) configured for HTTPS cloning; otherwise, the clone will fail. | 129 | - The build process clones the `msprof` and `msprobe` submodules via `git clone` (used for building the msprof analysis wheel and syncing the msaccucmp tool, respectively). These submodules are hosted on gitcode and require a [gitcode personal access token](https://gitcode.com/setting/token-classic) configured for HTTPS cloning; otherwise, the clone will fail. |
| 129 | -- If the build environment cannot access the network, refer to [Offline Build Environment Preparation](docs/en/quick_install.md#offline-build-environment-preparation) to complete the download and configuration of dependency packages in advance. Then specify the dependency package directory through the `--cann_3rd_lib_path` parameter before running the build. | 130 | +- If the build environment cannot access the network, refer to [Offline Build Environment Preparation](docs/en/quick_install.md#offline-build-environment-preparation) to complete the download and configuration of dependency packages in advance. Then specify the dependency package directory through the `--cann_3rd_lib_path` parameter before running the build. The offline prestaging script `cmake/download_libs.py` also supports `--bundle_branch` to select which branch's closed-source package to prestage (auto-detected by default); it must match the branch used at build time. |
| 130 | - Closed-source binary packages are extracted to `bundle/` in the repository root. If `bundle/` already exists and is non-empty, the build reuses it and skips downloading. To force a fresh download or recover from an incomplete `bundle/`, run `bash build.sh --make_clean` before rebuilding, or manually delete `bundle/` and run `bash build.sh` again. | 131 | - Closed-source binary packages are extracted to `bundle/` in the repository root. If `bundle/` already exists and is non-empty, the build reuses it and skips downloading. To force a fresh download or recover from an incomplete `bundle/`, run `bash build.sh --make_clean` before rebuilding, or manually delete `bundle/` and run `bash build.sh` again. |
| 131 | - For more build parameters, run `bash build.sh -h`. | 132 | - For more build parameters, run `bash build.sh -h`. |
| 132 | 133 | ||
| @@ -44,6 +44,9 @@ usage() { | |||
| 44 | echo " Set ascend package install path, default /usr/local/Ascend/cann" | 44 | echo " Set ascend package install path, default /usr/local/Ascend/cann" |
| 45 | echo " --cann_3rd_lib_path=<PATH>" | 45 | echo " --cann_3rd_lib_path=<PATH>" |
| 46 | echo " Set ascend third_party package install path, default ./third_party" | 46 | echo " Set ascend third_party package install path, default ./third_party" |
| 47 | + echo " --bundle_branch=<NAME>" | ||
| 48 | + echo " Set the branch whose closed-source bundle to download (e.g. master, 9.1.0)." | ||
| 49 | + echo " Default: auto-detected from git; falls back to master." | ||
| 47 | echo "Test Options:" | 50 | echo "Test Options:" |
| 48 | echo $dotted_line | 51 | echo $dotted_line |
| 49 | echo " -u Build and run all unit tests" | 52 | echo " -u Build and run all unit tests" |
| @@ -71,6 +74,9 @@ checkopts() { | |||
| 71 | TEST_COMPONENT="all" | 74 | TEST_COMPONENT="all" |
| 72 | RUN_UT_ONLY="off" | 75 | RUN_UT_ONLY="off" |
| 73 | RUN_ST_ONLY="off" | 76 | RUN_ST_ONLY="off" |
| 77 | + # 必须显式清空：否则环境里同名变量会被当成"用户显式指定"透传给 CMake， | ||
| 78 | + # 既盖掉配置期的 git 探测，又绕过下方 --bundle_branch 解析处的字符校验。 | ||
| 79 | + BUNDLE_BRANCH="" | ||
| 74 | 80 | ||
| 75 | if [[ -n "${ASCEND_HOME_PATH}" ]]; then | 81 | if [[ -n "${ASCEND_HOME_PATH}" ]]; then |
| 76 | echo "env exists ASCEND_HOME_PATH : ${ASCEND_HOME_PATH}" | 82 | echo "env exists ASCEND_HOME_PATH : ${ASCEND_HOME_PATH}" |
| @@ -83,7 +89,7 @@ checkopts() { | |||
| 83 | CANN_3RD_LIB_PATH="$BASEPATH/third_party" | 89 | CANN_3RD_LIB_PATH="$BASEPATH/third_party" |
| 84 | 90 | ||
| 85 | # Process the options | 91 | # Process the options |
| 86 | - parsed_args=$(getopt -a -o j:hvuO: -l help,verbose,cov,make_clean,build-type:,pkg-type:,noexec,ascend_install_path:,pkg,asan,cann_3rd_lib_path:,component:,ut,st -- "$@") || { | 92 | + parsed_args=$(getopt -a -o j:hvuO: -l help,verbose,cov,make_clean,build-type:,pkg-type:,noexec,ascend_install_path:,pkg,asan,cann_3rd_lib_path:,bundle_branch:,component:,ut,st -- "$@") || { |
| 87 | usage | 93 | usage |
| 88 | exit 1 | 94 | exit 1 |
| 89 | } | 95 | } |
| @@ -144,6 +150,17 @@ checkopts() { | |||
| 144 | CANN_3RD_LIB_PATH="$(realpath $2)" | 150 | CANN_3RD_LIB_PATH="$(realpath $2)" |
| 145 | shift 2 | 151 | shift 2 |
| 146 | ;; | 152 | ;; |
| 153 | + --bundle_branch) | ||
| 154 | + BUNDLE_BRANCH="$2" | ||
| 155 | + # 只允许 git ref 常见字符：CMAKE_ARGS 在 cmake ${cmake_args} .. 处是非引号展开， | ||
| 156 | + # 分支名若含空格/引号/shell 元字符会被二次解析。此处拒绝而非加引号包裹， | ||
| 157 | + # 因为包裹后引号会作为字面量传给 CMake（见下方拼装处注释）。 | ||
| 158 | + if [[ ! "${BUNDLE_BRANCH}" =~ ^[A-Za-z0-9._/-]+$ ]]; then | ||
| 159 | + echo "ERROR: invalid --bundle_branch '${BUNDLE_BRANCH}': only [A-Za-z0-9._/-] allowed." | ||
| 160 | + exit 1 | ||
| 161 | + fi | ||
| 162 | + shift 2 | ||
| 163 | + ;; | ||
| 147 | --asan) | 164 | --asan) |
| 148 | ENABLE_ASAN="on" | 165 | ENABLE_ASAN="on" |
| 149 | shift | 166 | shift |
| @@ -277,6 +294,13 @@ build_oam_tools() { | |||
| 277 | -DBUILD_OPEN_PROJECT=ON\ | 294 | -DBUILD_OPEN_PROJECT=ON\ |
| 278 | -DENABLE_PACKAGE=TRUE \ | 295 | -DENABLE_PACKAGE=TRUE \ |
| 279 | -DPACKAGE_TYPE=${PACKAGE_TYPE}" | 296 | -DPACKAGE_TYPE=${PACKAGE_TYPE}" |
| 297 | + # 仅在用户显式指定 --bundle_branch 时透传；不传则由 cmake 配置期 git 探测决定。 | ||
| 298 | + # 不给取值加引号：CMAKE_ARGS 在 cmake_generate_make 里是 cmake ${cmake_args} .. 非引号展开， | ||
| 299 | + # 引号会作为字面量进入 CMake（OAM_BUNDLE_BRANCH 变成 "9.1.0" 而非 9.1.0），白名单校验必然失配。 | ||
| 300 | + # 取值合法性已在 --bundle_branch 解析处校验，此处无需再防 shell 元字符。 | ||
| 301 | + if [ -n "${BUNDLE_BRANCH}" ]; then | ||
 J问题：BUNDLE_BRANCH 未在 checkopts 中初始化，环境变量可绕过 --bundle_branch 解析处的白名单校验。
  | |||
| 302 | + CMAKE_ARGS="${CMAKE_ARGS} -DOAM_BUNDLE_BRANCH=${BUNDLE_BRANCH}" | ||
| 303 | + fi | ||
| 280 | cmake_generate_make "${BUILD_PATH}" "${CMAKE_ARGS}" | 304 | cmake_generate_make "${BUILD_PATH}" "${CMAKE_ARGS}" |
| 281 | 305 | ||
| 282 | # make package 前清理历史产物，避免旧 cann*.run/rpm/deb 被误当本次产物搬走。 | 306 | # make package 前清理历史产物，避免旧 cann*.run/rpm/deb 被误当本次产物搬走。 |
| @@ -16,12 +16,118 @@ | |||
| 16 | # limitations under the License. | 16 | # limitations under the License. |
| 17 | # ---------------------------------------------------------------------------- | 17 | # ---------------------------------------------------------------------------- |
| 18 | import os | 18 | import os |
| 19 | +import re | ||
| 20 | +import sys | ||
| 21 | +import argparse | ||
| 19 | import logging | 22 | import logging |
| 20 | import subprocess | 23 | import subprocess |
| 21 | import shutil | 24 | import shutil |
| 22 | 25 | ||
| 23 | logging.basicConfig(level=logging.INFO) | 26 | logging.basicConfig(level=logging.INFO) |
| 24 | 27 | ||
| 28 | +# 闭源包(bundle) OBS 基址与分支配置，须与 cmake/install_bundle.cmake 保持一致： | ||
| 29 | +# 离线预置在此按分支拼出 bundle 下载地址，联编时 install_bundle.cmake 按同样规则解析分支。 | ||
| 30 | +OAM_BUNDLE_BASE_URL = ( | ||
| 31 | + "https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag" | ||
| 32 | +) | ||
| 33 | +# OBS 上实际提供包的分支白名单（与 install_bundle.cmake 的 OAM_BUNDLE_KNOWN_BRANCHES 同步）。 | ||
| 34 | +OAM_BUNDLE_KNOWN_BRANCHES = ("master", "9.1.0") | ||
| 35 | +# 发布线分支的识别模式（与 install_bundle.cmake 的 OAM_BUNDLE_RELEASE_REF_REGEX 同步）： | ||
| 36 | +# 形如 9.1.0 或 9.1.0-beta.3 的远端分支都算发布线。不硬编码具体 ref——同一条发布线上 | ||
| 37 | +# 并存 9.1.0、9.1.0-beta.1/2/3 等多个分支，硬编码任一个都会让其余分支探测不到而回退 master。 | ||
| 38 | +OAM_BUNDLE_RELEASE_REF_REGEX = re.compile(r"^\d+\.\d+\.\d+(-beta\.\d+)?$") | ||
| 39 | +# 需预置的目标架构；离线机器无法预知目标架构，两种都下以覆盖 x86_64 / aarch64 构建。 | ||
| 40 | +OAM_BUNDLE_ARCHES = ("x86_64", "aarch64") | ||
| 41 | + | ||
| 42 | + | ||
| 43 | +def map_ref_to_obs_branch(ref): | ||
| 44 | + """把远端 ref 归一化为 OBS 路径名；非发布线返回 None。 | ||
| 45 | + | ||
| 46 | + 去掉 remote 前缀后：master 原样；发布线去掉 -beta.N 后缀（9.1.0-beta.3 -> 9.1.0）。 | ||
| 47 | + """ | ||
| 48 | + name = ref.split("/", 1)[-1] | ||
| 49 | + if name == "master": | ||
| 50 | + return "master" | ||
| 51 | + if OAM_BUNDLE_RELEASE_REF_REGEX.match(name): | ||
| 52 | + return re.sub(r"-beta\.\d+$", "", name) | ||
| 53 | + return None | ||
| 54 | + | ||
| 55 | + | ||
| 56 | +def detect_bundle_branch(): | ||
| 57 | + """git 探测所属发布分支：枚举远端分支，取领先提交数最小者（血缘最近）。 | ||
| 58 | + | ||
| 59 | + 与 cmake/install_bundle.cmake 的 oam_resolve_bundle_branch 探测逻辑一致； | ||
| 60 | + git 不可用或无候选命中时返回 None，由调用方兜底 master。 | ||
| 61 | + """ | ||
| 62 | + git_path = shutil.which("git") | ||
| 63 | + if not git_path: | ||
| 64 | + return None | ||
| 65 | + listed = subprocess.run( | ||
| 66 | + [git_path, "for-each-ref", "--format=%(refname:short)", "refs/remotes"], | ||
| 67 | + capture_output=True, text=True, check=False | ||
| 68 | + ) | ||
| 69 | + if listed.returncode != 0: | ||
| 70 | + logging.warning("git for-each-ref refs/remotes failed: %s", listed.stderr.strip()) | ||
| 71 | + return None | ||
| 72 | + best_branch = None | ||
| 73 | + best_ahead = None | ||
| 74 | + for ref in listed.stdout.split(): | ||
| 75 | + mapped = map_ref_to_obs_branch(ref) | ||
| 76 | + # 只比较 OBS 上确有包的分支，避免探测出必然触发白名单报错的发布线。 | ||
| 77 | + if mapped is None or mapped not in OAM_BUNDLE_KNOWN_BRANCHES: | ||
| 78 | + continue | ||
| 79 | + count = subprocess.run( | ||
| 80 | + [git_path, "rev-list", "--count", f"{ref}..HEAD"], | ||
| 81 | + capture_output=True, text=True, check=False | ||
| 82 | + ) | ||
| 83 | + if count.returncode != 0: | ||
| 84 | + logging.warning("git rev-list --count %s..HEAD failed: %s", ref, count.stderr.strip()) | ||
| 85 | + continue | ||
| 86 | + ahead = int(count.stdout.strip() or "0") | ||
| 87 | + if best_ahead is None or ahead < best_ahead: | ||
| 88 | + best_ahead = ahead | ||
| 89 | + best_branch = mapped | ||
| 90 | + return best_branch | ||
| 91 | + | ||
| 92 | + | ||
| 93 | +def resolve_bundle_branch(explicit=None): | ||
| 94 | + """解析要预置的 bundle 分支：显式指定 > git 探测 > master 兜底。 | ||
| 95 | + | ||
| 96 | + 与 install_bundle.cmake 一致，并对结果做白名单硬校验，避免拼出无对应包的地址。 | ||
| 97 | + """ | ||
| 98 | + branch = explicit if explicit else (detect_bundle_branch() or "master") | ||
| 99 | + if branch not in OAM_BUNDLE_KNOWN_BRANCHES: | ||
| 100 | + known = ", ".join(OAM_BUNDLE_KNOWN_BRANCHES) | ||
| 101 | + raise ValueError( | ||
| 102 | + f"bundle branch '{branch}' has no published package on OBS; known branches: {known}" | ||
| 103 | + ) | ||
| 104 | + return branch | ||
| 105 | + | ||
| 106 | + | ||
| 107 | +def bundle_urls(branch): | ||
| 108 | + """按分支拼各架构的 bundle 下载地址（下载路径恒为 release 包）。""" | ||
| 109 | + return [ | ||
| 110 | + (f"{OAM_BUNDLE_BASE_URL}/{branch}/cann-oam-tools-release-{arch}.tar.gz",) | ||
| 111 | + for arch in OAM_BUNDLE_ARCHES | ||
| 112 | + ] | ||
| 113 | + | ||
| 114 | + | ||
| 115 | +def write_bundle_branch_metadata(branch, tar_paths): | ||
| 116 | + """在预置包旁写 <tar>.branch 元数据，记录该包来自哪个分支。 | ||
| 117 | + | ||
| 118 | + 预置包文件名不含分支信息（各分支同名），联编时无从分辨。install_bundle.cmake | ||
| 119 | + 命中本地预置包后会读此文件校验分支，不匹配则在配置阶段报错， | ||
| 120 | + 避免离线/缓存构建静默混入其它分支的闭源包。 | ||
| 121 | + | ||
| 122 | + tar_paths 只应传本轮确实下载成功的包：仅按文件是否存在来写，会把目录里 | ||
| 123 | + 残留的旧分支 tar 误标成本轮分支，联编时反而错误通过校验。 | ||
| 124 | + """ | ||
| 125 | + for tar_path in tar_paths: | ||
| 126 | + meta_path = f"{tar_path}.branch" | ||
| 127 | + with open(meta_path, "w", encoding="utf-8") as meta_file: | ||
| 128 | + meta_file.write(f"{branch}\n") | ||
| 129 | + logging.info("Wrote bundle branch metadata %s -> %s", meta_path, branch) | ||
| 130 | + | ||
| 25 | 131 | ||
| 26 | def download_single_file(url, current_dir): | 132 | def download_single_file(url, current_dir): |
| 27 | """下载单个文件""" | 133 | """下载单个文件""" |
| @@ -52,7 +158,7 @@ def download_single_file(url, current_dir): | |||
| 52 | if result.returncode != 0: | 158 | if result.returncode != 0: |
| 53 | raise RuntimeError(f"git clone failed: {result.stderr}") | 159 | raise RuntimeError(f"git clone failed: {result.stderr}") |
| 54 | logging.info("Successfully cloned to %s", repo_path) | 160 | logging.info("Successfully cloned to %s", repo_path) |
| 55 | - return | 161 | + return repo_path |
| 56 | 162 | ||
| 57 | file_name = custom_name if custom_name else actual_url.split("/")[-1] | 163 | file_name = custom_name if custom_name else actual_url.split("/")[-1] |
| 58 | if not file_name: | 164 | if not file_name: |
| @@ -75,21 +181,44 @@ def download_single_file(url, current_dir): | |||
| 75 | check=False | 181 | check=False |
| 76 | ) | 182 | ) |
| 77 | if result.returncode != 0: | 183 | if result.returncode != 0: |
| 184 | + # wget -O 会先建目标文件，失败时残留半截内容；留着会被后续误当作可用预置包。 | ||
| 185 | + if os.path.exists(file_path): | ||
| 186 | + os.remove(file_path) | ||
| 78 | raise RuntimeError(f"wget download failed: {result.stderr}") | 187 | raise RuntimeError(f"wget download failed: {result.stderr}") |
| 79 | logging.info("Successfully saved to %s", file_path) | 188 | logging.info("Successfully saved to %s", file_path) |
| 189 | + return file_path | ||
| 80 | 190 | ||
| 81 | 191 | ||
| 82 | def download_files_native(url_list): | 192 | def download_files_native(url_list): |
| 83 | - """下载多个文件""" | 193 | + """下载多个文件，返回本轮成功落盘的路径集合。 |
| 194 | + | ||
| 195 | + 单个失败不中断其余下载（保持原有行为），但失败者不进返回集合—— | ||
| 196 | + 调用方据此只给本轮确实取到的包写元数据，避免把目录里的旧包误标成新分支。 | ||
| 197 | + """ | ||
| 84 | current_dir = os.getcwd() | 198 | current_dir = os.getcwd() |
| 199 | + downloaded = set() | ||
| 85 | for url in url_list: | 200 | for url in url_list: |
| 86 | try: | 201 | try: |
| 87 | - download_single_file(url, current_dir) | 202 | + downloaded.add(download_single_file(url, current_dir)) |
| 88 | except (ValueError, RuntimeError, OSError) as e: | 203 | except (ValueError, RuntimeError, OSError) as e: |
| 89 | logging.info("Download file form %s failed: %s", url, e) | 204 | logging.info("Download file form %s failed: %s", url, e) |
| 205 | + return downloaded | ||
| 90 | 206 | ||
| 91 | 207 | ||
| 92 | if __name__ == "__main__": | 208 | if __name__ == "__main__": |
| 209 | + parser = argparse.ArgumentParser( | ||
| 210 | + description="离线预置 oam-tools 编译依赖（第三方库 + 闭源 bundle 包 + 子仓）") | ||
| 211 | + parser.add_argument( | ||
| 212 | + "--bundle_branch", default=None, | ||
| 213 | + help="预置哪个分支的闭源 bundle 包（可选 master / 9.1.0）。" | ||
| 214 | + "不指定则按当前 git 提交自动探测所属发布分支，探测不出时回退 master。" | ||
| 215 | + "须与联编时 build.sh --bundle_branch / install_bundle.cmake 的解析结果一致。") | ||
| 216 | + args = parser.parse_args() | ||
| 217 | + | ||
| 218 | + # bundle 分支解析与 install_bundle.cmake 同规则：显式 > git 探测 > master 兜底 + 白名单校验。 | ||
| 219 | + bundle_branch = resolve_bundle_branch(args.bundle_branch) | ||
| 220 | + logging.info("prefetch bundle for branch: %s", bundle_branch) | ||
| 221 | + | ||
| 93 | my_urls = [ | 222 | my_urls = [ |
| 94 | ( | 223 | ( |
| 95 | "https://gitcode.com/cann-src-third-party/protobuf/releases/download/v25.1" | 224 | "https://gitcode.com/cann-src-third-party/protobuf/releases/download/v25.1" |
| @@ -115,20 +244,41 @@ if __name__ == "__main__": | |||
| 115 | "https://gitcode.com/cann-src-third-party/mockcpp/releases/download" | 244 | "https://gitcode.com/cann-src-third-party/mockcpp/releases/download" |
| 116 | "/v2.7-h2/mockcpp-2.7.tar.gz", | 245 | "/v2.7-h2/mockcpp-2.7.tar.gz", |
| 117 | ), | 246 | ), |
| 118 | - ( | 247 | + ] |
| 119 | - "https://cann-3rd.obs.cn-north-4.myhuaweicloud.com" | 248 | + # 闭源 bundle 包按解析出的分支拼地址（各架构一份），不再硬编码 master。 |
| 120 | - "/cann/oam-tools-diag/master/cann-oam-tools-release-x86_64.tar.gz", | 249 | + bundle_url_list = bundle_urls(bundle_branch) |
| 121 | - ), | 250 | + my_urls.extend(bundle_url_list) |
| 122 | - ( | 251 | + my_urls.extend([ |
| 123 | - "https://cann-3rd.obs.cn-north-4.myhuaweicloud.com" | ||
| 124 | - "/cann/oam-tools-diag/master/cann-oam-tools-release-aarch64.tar.gz", | ||
| 125 | - ), | ||
| 126 | ( | 252 | ( |
| 127 | "https://gitcode.com/Ascend/msprobe.git", | 253 | "https://gitcode.com/Ascend/msprobe.git", |
| 128 | ), | 254 | ), |
| 129 | ( | 255 | ( |
| 130 | "https://gitcode.com/Ascend/msprof.git", | 256 | "https://gitcode.com/Ascend/msprof.git", |
| 131 | ), | 257 | ), |
| 132 | - ] | 258 | + ]) |
| 133 | 259 | ||
| 134 | - download_files_native(my_urls) | 260 | + downloaded = download_files_native(my_urls) |
| 261 | + # 预置包名不含分支，旁写元数据供联编时校验，避免混用其它分支的闭源包。 | ||
| 262 | + # 只给本轮确实下载成功的 bundle tar 写：否则目录里的旧分支包会被误标成本轮分支， | ||
| 263 | + # 联编时反而错误通过校验。 | ||
| 264 | + bundle_tars = [ | ||
| 265 | + os.path.join(os.getcwd(), url[0].split("/")[-1]) | ||
| 266 | + for url in bundle_url_list | ||
| 267 | + ] | ||
| 268 | + fetched_bundle_tars = [tar for tar in bundle_tars if tar in downloaded] | ||
| 269 | + if not fetched_bundle_tars: | ||
| 270 | + logging.error( | ||
| 271 | + "no bundle tarball downloaded for branch %s; " | ||
| 272 | + "offline build cannot use prestaged bundle", bundle_branch) | ||
| 273 | + sys.exit(1) | ||
| 274 | + write_bundle_branch_metadata(bundle_branch, fetched_bundle_tars) | ||
| 275 | + # 本轮没取到的包若还留着上一轮的元数据，会被联编时当成"已核对"而错误放行； | ||
| 276 | + # 删掉元数据让 install_bundle.cmake 走"无元数据"告警分支，而非错误通过校验。 | ||
| 277 | + for tar in bundle_tars: | ||
| 278 | + if tar in downloaded: | ||
| 279 | + continue | ||
| 280 | + logging.warning("bundle tarball not fetched this run: %s", tar) | ||
| 281 | + stale_meta = f"{tar}.branch" | ||
| 282 | + if os.path.exists(stale_meta): | ||
| 283 | + os.remove(stale_meta) | ||
| 284 | + logging.warning("removed stale branch metadata %s", stale_meta) | ||
| @@ -16,13 +16,120 @@ | |||
| 16 | 16 | ||
| 17 | # 闭源包(bundle)拉取与解压：原 install_bundle.sh 的 CMake 版本。 | 17 | # 闭源包(bundle)拉取与解压：原 install_bundle.sh 的 CMake 版本。 |
| 18 | # 按 架构 + 分支 拼出 OBS 地址，下载后解压到 ${OAM_TOOLS_DIR}/bundle。 | 18 | # 按 架构 + 分支 拼出 OBS 地址，下载后解压到 ${OAM_TOOLS_DIR}/bundle。 |
| 19 | -# 分支固定 master —— 功能分支没有对应的已发布包。 | ||
| 20 | -set(OAM_BUNDLE_BRANCH "master") | ||
| 21 | set(OAM_BUNDLE_BASE_URL | 19 | set(OAM_BUNDLE_BASE_URL |
| 22 | "https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag") | 20 | "https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag") |
| 23 | 21 | ||
| 22 | +# OBS 上实际存在包的分支白名单。新增 release 线时须在此同步， | ||
| 23 | +# 并先确认 OBS bucket 下已上传对应的 <branch>/cann-oam-tools-release-<arch>.tar.gz。 | ||
| 24 | +# （已核实：截至当前仅 master、9.1.0 两条路径可下载，其余返回 403。） | ||
| 25 | +set(OAM_BUNDLE_KNOWN_BRANCHES "master" "9.1.0") | ||
| 26 | + | ||
| 27 | +# 记录 bundle/ 内容来自哪个分支。解压产物本身不含分支信息，切分支后复用旧 bundle | ||
| 28 | +# 无从分辨，故取包成功后在此落一份元数据，下次命中已有 bundle 时据它校验。 | ||
| 29 | +set(OAM_BUNDLE_META_NAME ".bundle_branch") | ||
| 30 | + | ||
| 31 | +# 发布线分支的识别模式：形如 9.1.0 或 9.1.0-beta.3 的远端分支都算发布线。 | ||
| 32 | +# 不硬编码具体 ref——同一条发布线上并存 9.1.0、9.1.0-beta.1/2/3 等多个分支， | ||
| 33 | +# 硬编码任一个都会让其余分支探测不到而静默回退 master。 | ||
| 34 | +set(OAM_BUNDLE_RELEASE_REF_REGEX "^[0-9]+\\.[0-9]+\\.[0-9]+(-beta\\.[0-9]+)?$") | ||
| 35 | + | ||
级别：提示 问题：OAM_BUNDLE_REF_CANDIDATES 硬编码了 origin/9.1.0-beta.2 这一实际 git 分支名，与 OBS 路径名 9.1.0 不一致需手动映射。 影响：若 9.1.0 线分支重命名或新增 release 线使用不同命名规范，探测会静默失败并回退 master，拉到错误分支的包。 修复建议：考虑在注释或 CI 中增加校验，定期确认 ref 名与远端实际分支一致；或在探测失败时输出 WARNING 级别日志提示哪些 ref 未命中。 | |||
| 36 | +# 解析要拉取的 bundle 分支：显式指定 > git 探测(领先提交数最小) > master 兜底。 | ||
| 37 | +# 结果写入 RESULT_VAR（PARENT_SCOPE）。仅在确需从 OBS 下载时调用， | ||
| 38 | +# 避免"bundle 已就绪 / 命中本地预置包"时做无谓的 git 探测或误触白名单校验。 | ||
| 39 | +function(oam_resolve_bundle_branch RESULT_VAR) | ||
| 40 | + # 1) 显式 -DOAM_BUNDLE_BRANCH 优先。 | ||
| 41 | + if(DEFINED OAM_BUNDLE_BRANCH AND NOT OAM_BUNDLE_BRANCH STREQUAL "") | ||
| 42 | + message(STATUS "bundle branch (explicit): ${OAM_BUNDLE_BRANCH}") | ||
级别：一般 问题：oam_resolve_bundle_branch 新增了显式指定、git 探测、兜底三条路径及白名单校验，但无任何测试覆盖。 影响：git 探测逻辑（ahead-count 最小优先）在浅克隆、HEAD 为多个 release 分支祖先、ref 不存在等边界场景下行为无法验证，后续维护易引入回归。 修复建议：补充 CMake 测试或 CI 集成用例，至少覆盖：显式指定命中/未命中白名单、git 不可用回退、HEAD 在 9.1.0 线上探测到 9.1.0、HEAD 在 master 上探测到 master、两个 ref 均不存在回退 master。 | |||
| 43 | + set(${RESULT_VAR} "${OAM_BUNDLE_BRANCH}" PARENT_SCOPE) | ||
| 44 | + return() | ||
| 45 | + endif() | ||
| 46 | + | ||
| 47 | + # 2) git 探测：对每个存在的候选 ref 算 HEAD 相对其分家点的领先提交数， | ||
| 48 | + # 取最小者（血缘最近）。git 不可用或无候选命中则回退。 | ||
| 49 | + set(_best_branch "") | ||
| 50 | + set(_best_ahead "") | ||
| 51 | + # 自行确保 GIT_EXECUTABLE 可用，不依赖其他 cmake 文件的 include 顺序副作用 | ||
| 52 | + # （当前 build_submodules.cmake 恰在本文件之前 include 并已 find_package(Git)， | ||
| 53 | + # 但那属实现细节，顺序一变本函数就会静默跳过探测）。找不到 git 时走 master 兜底。 | ||
| 54 | + if(NOT GIT_EXECUTABLE) | ||
| 55 | + find_package(Git QUIET) | ||
| 56 | + endif() | ||
| 57 | + if(NOT GIT_EXECUTABLE) | ||
| 58 | + find_program(GIT_EXECUTABLE git) | ||
| 59 | + endif() | ||
| 60 | + if(GIT_EXECUTABLE) | ||
J 问题：
| |||
| 61 | + # 枚举已有的远端分支（各 remote 都算，fork 场景下 release 线常在 upstream）， | ||
| 62 | + # 再按模式筛出发布线。这样同一条线上的 9.1.0 / 9.1.0-beta.N 都能命中， | ||
| 63 | + # 不必逐个硬编码 ref。 | ||
| 64 | + execute_process( | ||
| 65 | + # --format 必须加引号：括号在 CMake 非引号实参里会被当作参数分隔， | ||
| 66 | + # 未加引号时该实参会被截成 "%"，git 只输出 "%" 而非分支名。 | ||
| 67 | + COMMAND ${GIT_EXECUTABLE} for-each-ref "--format=%(refname:short)" refs/remotes | ||
| 68 | + WORKING_DIRECTORY "${OAM_TOOLS_DIR}" | ||
| 69 | + RESULT_VARIABLE _ls_result | ||
| 70 | + OUTPUT_VARIABLE _ls_out | ||
| 71 | + OUTPUT_STRIP_TRAILING_WHITESPACE | ||
| 72 | + ERROR_VARIABLE _ls_err) | ||
| 73 | + if(NOT _ls_result EQUAL 0) | ||
| 74 | + message(WARNING | ||
| 75 | + "bundle branch detect: 'git for-each-ref refs/remotes' failed " | ||
| 76 | + "(code ${_ls_result}): ${_ls_err}; fall back to master") | ||
| 77 | + set(_ls_out "") | ||
| 78 | + endif() | ||
| 79 | + string(REPLACE "\n" ";" _remote_refs "${_ls_out}") | ||
| 80 | + foreach(_ref IN LISTS _remote_refs) | ||
| 81 | + # 去掉 remote 名前缀（origin/ 或 upstream/ 等），只看分支名本身。 | ||
| 82 | + string(REGEX REPLACE "^[^/]+/" "" _name "${_ref}") | ||
| 83 | + # 归一化成 OBS 路径名：master 原样；发布线去掉 -beta.N 后缀（9.1.0-beta.3 -> 9.1.0）。 | ||
| 84 | + if(_name STREQUAL "master") | ||
| 85 | + set(_mapped "master") | ||
| 86 | + elseif(_name MATCHES "${OAM_BUNDLE_RELEASE_REF_REGEX}") | ||
| 87 | + string(REGEX REPLACE "-beta\\.[0-9]+$" "" _mapped "${_name}") | ||
| 88 | + else() | ||
| 89 | + continue() | ||
| 90 | + endif() | ||
| 91 | + # 只保留 OBS 上确有包的分支作为候选：其余发布线（如尚未上传包的 9.2.0） | ||
| 92 | + # 直接忽略而非纳入比较，避免探测出一个必然触发白名单报错的分支。 | ||
| 93 | + if(NOT _mapped IN_LIST OAM_BUNDLE_KNOWN_BRANCHES) | ||
| 94 | + continue() | ||
| 95 | + endif() | ||
级别：提示 问题：git rev-list --count 失败时仅 continue() 且 ERROR_QUIET，无任何诊断信息。 影响：当 git 仓库损坏、权限不足或 ref 格式异常时，探测静默失败回退 master，用户无法区分'正常回退'与'git 异常导致误回退'。 修复建议：在 _cnt_result 非零时增加 message(WARNING ...) 输出 ref 名和错误码，便于排查。 | |||
| 96 | + # 领先提交数 = merge-base(HEAD, ref)..HEAD 的提交数。 | ||
| 97 | + execute_process( | ||
| 98 | + COMMAND ${GIT_EXECUTABLE} rev-list --count "${_ref}..HEAD" | ||
| 99 | + WORKING_DIRECTORY "${OAM_TOOLS_DIR}" | ||
| 100 | + RESULT_VARIABLE _cnt_result | ||
| 101 | + OUTPUT_VARIABLE _ahead | ||
| 102 | + OUTPUT_STRIP_TRAILING_WHITESPACE | ||
| 103 | + ERROR_VARIABLE _cnt_err) | ||
| 104 | + if(NOT _cnt_result EQUAL 0) | ||
| 105 | + # ref 已存在却仍数不出提交数，属异常（仓库损坏 / ref 格式异常等）。 | ||
| 106 | + # 输出 WARNING 区分"正常回退"与"git 异常误回退"，便于排查。 | ||
| 107 | + message(WARNING | ||
| 108 | + "bundle branch detect: 'git rev-list --count ${_ref}..HEAD' failed " | ||
| 109 | + "(code ${_cnt_result}): ${_cnt_err}; skip this candidate") | ||
| 110 | + continue() | ||
| 111 | + endif() | ||
| 112 | + if(_best_ahead STREQUAL "" OR _ahead LESS _best_ahead) | ||
| 113 | + set(_best_ahead "${_ahead}") | ||
| 114 | + set(_best_branch "${_mapped}") | ||
| 115 | + endif() | ||
| 116 | + endforeach() | ||
| 117 | + endif() | ||
| 118 | + | ||
| 119 | + if(NOT _best_branch STREQUAL "") | ||
| 120 | + message(STATUS "bundle branch (git-detected): ${_best_branch} (ahead ${_best_ahead})") | ||
| 121 | + set(${RESULT_VAR} "${_best_branch}" PARENT_SCOPE) | ||
| 122 | + return() | ||
| 123 | + endif() | ||
| 124 | + | ||
| 125 | + # 3) 兜底 master，保持无 git 环境 / CI 上的原有行为。 | ||
| 126 | + message(STATUS "bundle branch (fallback): master") | ||
| 127 | + set(${RESULT_VAR} "master" PARENT_SCOPE) | ||
| 128 | +endfunction() | ||
| 129 | + | ||
| 24 | function(oam_install_bundle) | 130 | function(oam_install_bundle) |
| 25 | set(_bundle_dir "${OAM_TOOLS_DIR}/bundle") | 131 | set(_bundle_dir "${OAM_TOOLS_DIR}/bundle") |
| 132 | + set(_bundle_meta "${_bundle_dir}/${OAM_BUNDLE_META_NAME}") | ||
| 26 | 133 | ||
| 27 | # bundle 已存在且非空即视为就绪，跳过下载（--make_clean 会先删掉它以强制刷新）。 | 134 | # bundle 已存在且非空即视为就绪，跳过下载（--make_clean 会先删掉它以强制刷新）。 |
| 28 | # 空目录（上一轮下载失败留下的残壳）需重新拉取，否则会带空 bundle 继续、 | 135 | # 空目录（上一轮下载失败留下的残壳）需重新拉取，否则会带空 bundle 继续、 |
| @@ -30,7 +137,31 @@ function(oam_install_bundle) | |||
| 30 | if(EXISTS "${_bundle_dir}" AND IS_DIRECTORY "${_bundle_dir}") | 137 | if(EXISTS "${_bundle_dir}" AND IS_DIRECTORY "${_bundle_dir}") |
| 31 | file(GLOB _bundle_entries "${_bundle_dir}/*") | 138 | file(GLOB _bundle_entries "${_bundle_dir}/*") |
| 32 | if(_bundle_entries) | 139 | if(_bundle_entries) |
| 33 | - message(STATUS "bundle already present at ${_bundle_dir}, skip download") | 140 | + # 复用已有 bundle 前必须核对它来自哪个分支：从 master 构建后切到 9.1.0 |
| 141 | + # 直接 build.sh 会命中这里，若不校验就会继续混用 master 的闭源包， | ||
| 142 | + # 这正是本改动要修的场景（连显式 --bundle_branch 也会被静默忽略）。 | ||
| 143 | + oam_resolve_bundle_branch(_bundle_branch) | ||
| 144 | + if(EXISTS "${_bundle_meta}") | ||
| 145 | + file(READ "${_bundle_meta}" _present_branch) | ||
| 146 | + string(STRIP "${_present_branch}" _present_branch) | ||
| 147 | + if(NOT _present_branch STREQUAL _bundle_branch) | ||
| 148 | + message(FATAL_ERROR | ||
| 149 | + "existing bundle at ${_bundle_dir} is from branch " | ||
| 150 | + "'${_present_branch}' but this build targets '${_bundle_branch}'.\n" | ||
| 151 | + "re-run build.sh --make_clean to refetch, " | ||
| 152 | + "or pass --bundle_branch=${_present_branch} to keep using it.") | ||
| 153 | + endif() | ||
| 154 | + message(STATUS | ||
| 155 | + "bundle already present at ${_bundle_dir} (branch ${_present_branch}), " | ||
| 156 | + "skip download") | ||
| 157 | + else() | ||
| 158 | + # 本改动之前拉下的 bundle 没有元数据，无从核对；告警而非报错， | ||
| 159 | + # 避免让既有工作目录必须先 --make_clean 才能继续构建。 | ||
| 160 | + message(WARNING | ||
| 161 | + "existing bundle at ${_bundle_dir} has no ${OAM_BUNDLE_META_NAME} " | ||
| 162 | + "metadata; cannot verify it matches target branch " | ||
| 163 | + "'${_bundle_branch}'. run build.sh --make_clean to refetch if unsure.") | ||
| 164 | + endif() | ||
| 34 | return() | 165 | return() |
| 35 | endif() | 166 | endif() |
| 36 | message(STATUS "bundle dir empty, re-downloading") | 167 | message(STATUS "bundle dir empty, re-downloading") |
| @@ -54,7 +185,6 @@ function(oam_install_bundle) | |||
| 54 | # 与原脚本 URL 恒为 cann-oam-tools-release-${ARCH} 的行为一致。 | 185 | # 与原脚本 URL 恒为 cann-oam-tools-release-${ARCH} 的行为一致。 |
| 55 | set(_local_tar_name "cann-oam-tools-${_build_type}-${CMAKE_SYSTEM_PROCESSOR}.tar.gz") | 186 | set(_local_tar_name "cann-oam-tools-${_build_type}-${CMAKE_SYSTEM_PROCESSOR}.tar.gz") |
| 56 | set(_release_tar_name "cann-oam-tools-release-${CMAKE_SYSTEM_PROCESSOR}.tar.gz") | 187 | set(_release_tar_name "cann-oam-tools-release-${CMAKE_SYSTEM_PROCESSOR}.tar.gz") |
| 57 | - set(_url "${OAM_BUNDLE_BASE_URL}/${OAM_BUNDLE_BRANCH}/${_release_tar_name}") | ||
| 58 | 188 | ||
| 59 | file(MAKE_DIRECTORY "${_bundle_dir}") | 189 | file(MAKE_DIRECTORY "${_bundle_dir}") |
| 60 | 190 | ||
| @@ -69,11 +199,55 @@ function(oam_install_bundle) | |||
| 69 | endif() | 199 | endif() |
| 70 | endforeach() | 200 | endforeach() |
| 71 | 201 | ||
| 202 | + # 取包来源的分支；仅在确知时才落 bundle/ 元数据，避免把"来源不明"记成目标分支。 | ||
| 203 | + set(_bundle_branch "") | ||
| 204 | + | ||
| 72 | if(NOT _local_tar STREQUAL "") | 205 | if(NOT _local_tar STREQUAL "") |
| 206 | + # 预置包文件名不含分支信息（各分支同名），直接复用会静默混入其它分支的闭源包。 | ||
| 207 | + # download_libs.py 会在包旁写 <tar>.branch 元数据，此处校验其与本次目标分支一致。 | ||
| 208 | + # 无元数据者按旧约定放行（兼容手工预置的既有包），仅告警提示无法核对。 | ||
| 209 | + if(EXISTS "${_local_tar}.branch") | ||
| 210 | + file(READ "${_local_tar}.branch" _local_tar_branch) | ||
| 211 | + string(STRIP "${_local_tar_branch}" _local_tar_branch) | ||
| 212 | + oam_resolve_bundle_branch(_bundle_branch) | ||
| 213 | + if(NOT _local_tar_branch STREQUAL _bundle_branch) | ||
| 214 | + message(FATAL_ERROR | ||
| 215 | + "local bundle tarball branch mismatch: ${_local_tar} is from " | ||
| 216 | + "'${_local_tar_branch}' but this build targets '${_bundle_branch}'.\n" | ||
| 217 | + "re-run cmake/download_libs.py --bundle_branch=${_bundle_branch}, " | ||
| 218 | + "or pass --bundle_branch=${_local_tar_branch} to match the prestaged package.") | ||
| 219 | + endif() | ||
| 220 | + message(STATUS "bundle local tarball branch verified: ${_local_tar_branch}") | ||
| 221 | + else() | ||
| 222 | + message(WARNING | ||
| 223 | + "local bundle tarball ${_local_tar} has no .branch metadata; " | ||
| 224 | + "cannot verify it matches the target branch. " | ||
| 225 | + "re-run cmake/download_libs.py to generate it.") | ||
| 226 | + endif() | ||
| 73 | message(STATUS "bundle using local tarball: ${_local_tar}") | 227 | message(STATUS "bundle using local tarball: ${_local_tar}") |
| 74 | file(COPY "${_local_tar}" DESTINATION "${_bundle_dir}") | 228 | file(COPY "${_local_tar}" DESTINATION "${_bundle_dir}") |
| 75 | set(_tar_path "${_bundle_dir}/${_local_tar_name}") | 229 | set(_tar_path "${_bundle_dir}/${_local_tar_name}") |
J 问题：本地预置包命中时不会解析或校验 bundle 分支，且预置包文件名不包含分支信息。用户即使显式传入 --bundle_branch=9.1.0，只要 CANN_3RD_LIB_PATH 中已有同名的 master release tar，构建就会直接复制该包并跳过分支解析。
| |||
| 76 | else() | 230 | else() |
| 231 | + # 仅在确需下载时才解析分支：bundle 已就绪或命中本地预置包时不会走到这里， | ||
| 232 | + # 从而避免无谓的 git 探测，以及"有缓存却因分支校验失败"的误报。 | ||
| 233 | + oam_resolve_bundle_branch(_bundle_branch) | ||
| 234 | + # 白名单硬校验：拼出的分支若 OBS 上没有对应包，立即报错， | ||
| 235 | + # 避免下载到 403 空包后把失败推迟到后续 install() 阶段更难定位。 | ||
| 236 | + if(NOT _bundle_branch IN_LIST OAM_BUNDLE_KNOWN_BRANCHES) | ||
| 237 | + string(REPLACE ";" ", " _known "${OAM_BUNDLE_KNOWN_BRANCHES}") | ||
| 238 | + # 提示里逐个列出 --bundle_branch=<name>，不用 <...> 包裹取值列表： | ||
| 239 | + # 尖括号是 shell 重定向元字符，用户整行复制到终端会被解析为重定向而报错。 | ||
| 240 | + set(_hint "") | ||
| 241 | + foreach(_b IN LISTS OAM_BUNDLE_KNOWN_BRANCHES) | ||
| 242 | + string(APPEND _hint " --bundle_branch=${_b}") | ||
| 243 | + endforeach() | ||
| 244 | + message(FATAL_ERROR | ||
| 245 | + "bundle branch '${_bundle_branch}' has no published package on OBS.\n" | ||
| 246 | + "known branches: ${_known}\n" | ||
| 247 | + "specify one explicitly, e.g.${_hint}") | ||
| 248 | + endif() | ||
| 249 | + set(_url "${OAM_BUNDLE_BASE_URL}/${_bundle_branch}/${_release_tar_name}") | ||
| 250 | + | ||
| 77 | # 下载路径只提供 release 包；debug 构建若未预置本地 debug 包，将回退到 release bundle。 | 251 | # 下载路径只提供 release 包；debug 构建若未预置本地 debug 包，将回退到 release bundle。 |
| 78 | set(_tar_path "${_bundle_dir}/${_release_tar_name}") | 252 | set(_tar_path "${_bundle_dir}/${_release_tar_name}") |
| 79 | message(STATUS "bundle download (release only): ${_url}") | 253 | message(STATUS "bundle download (release only): ${_url}") |
| @@ -97,6 +271,12 @@ function(oam_install_bundle) | |||
| 97 | 271 | ||
| 98 | # 解压后删除 tar 包，保持 bundle 目录只含解压产物。 | 272 | # 解压后删除 tar 包，保持 bundle 目录只含解压产物。 |
| 99 | file(REMOVE "${_tar_path}") | 273 | file(REMOVE "${_tar_path}") |
| 274 | + | ||
| 275 | + # 落分支元数据供下次复用时校验。仅在确知来源分支时写：命中本地预置包但其无 | ||
| 276 | + # .branch 元数据时 _bundle_branch 为空，此时来源不明，不能记成目标分支。 | ||
| 277 | + if(NOT _bundle_branch STREQUAL "") | ||
| 278 | + file(WRITE "${_bundle_meta}" "${_bundle_branch}\n") | ||
| 279 | + endif() | ||
| 100 | message(STATUS "bundle ready at ${_bundle_dir}") | 280 | message(STATUS "bundle ready at ${_bundle_dir}") |
| 101 | endfunction() | 281 | endfunction() |
| 102 | 282 | ||
| @@ -253,6 +253,16 @@ Run the download script in a networked environment. This script directly downloa | |||
| 253 | python cmake/download_libs.py | 253 | python cmake/download_libs.py |
| 254 | ``` | 254 | ``` |
| 255 | 255 | ||
| 256 | +Closed-source binary packages are fetched per branch. By default the script detects the release branch the current git commit belongs to (falling back to `master` when detection fails), using the same branch resolution rules as `build.sh` / `install_bundle.cmake` at build time. To prestage a specific branch, specify it explicitly with `--bundle_branch` (currently `master` or `9.1.0`; any other value fails): | ||
| 257 | + | ||
| 258 | +```bash | ||
| 259 | +# Explicitly prestage the closed-source package for the 9.1.0 branch; | ||
| 260 | +# must match the later build with bash build.sh --bundle_branch=9.1.0 | ||
| 261 | +python cmake/download_libs.py --bundle_branch=9.1.0 | ||
| 262 | +``` | ||
| 263 | + | ||
| 264 | +> Note: The prestaged bundle branch must match the branch resolved at build time in the target environment; otherwise the build fails because the package for that branch is missing locally. Detection may be inaccurate for personal branches, so specifying `--bundle_branch` explicitly for both download and build is recommended. | ||
| 265 | + | ||
| 256 | ### Upload Dependency Packages | 266 | ### Upload Dependency Packages |
| 257 | 267 | ||
| 258 | Create a new `third_party_path` directory in the build environment to store third-party open source software and closed-source software | 268 | Create a new `third_party_path` directory in the build environment to store third-party open source software and closed-source software |
| @@ -275,7 +285,9 @@ Third-party libraries, closed-source binary packages, and subrepositories includ | |||
| 275 | |mockcpp-patch|2.7-h2|[mockcpp-2.7_py3.patch](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7_py3.patch)| | 285 | |mockcpp-patch|2.7-h2|[mockcpp-2.7_py3.patch](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7_py3.patch)| |
| 276 | |mockcpp|2.7-h2|[mockcpp-2.7.tar.gz](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7.tar.gz)| | 286 | |mockcpp|2.7-h2|[mockcpp-2.7.tar.gz](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7.tar.gz)| |
| 277 | 287 | ||
| 278 | -| Closed-source Binary | Version | Download Address | | 288 | +Closed-source binary packages are fetched per branch. The table below uses the default `master` branch as an example; to prestage another branch, replace `master` in the address with the corresponding branch name (for example `9.1.0`), or run `python cmake/download_libs.py --bundle_branch=<branch>` to download the package for that branch automatically. |
| 289 | + | ||
| 290 | +| Closed-source Binary | Branch | Download Address | | ||
| 279 | |---|---|---| | 291 | |---|---|---| |
| 280 | |cann-oam-tools-release-x86_64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-x86_64.tar.gz)| | 292 | |cann-oam-tools-release-x86_64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-x86_64.tar.gz)| |
| 281 | |cann-oam-tools-release-aarch64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-aarch64.tar.gz)| | 293 | |cann-oam-tools-release-aarch64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-aarch64.tar.gz)| |
| @@ -255,6 +255,15 @@ git clone -b ${branch} https://gitcode.com/cann/oam-tools.git | |||
| 255 | python cmake/download_libs.py | 255 | python cmake/download_libs.py |
| 256 | ``` | 256 | ``` |
| 257 | 257 | ||
| 258 | +闭源二进制包按分支拉取，脚本默认根据当前 git 提交自动探测所属发布分支（探测不出时回退 `master`），与联编时 `build.sh` / `install_bundle.cmake` 的分支解析规则一致。如需预置指定分支的包，用 `--bundle_branch` 显式指定（当前可选 `master`、`9.1.0`，其它值会报错）： | ||
| 259 | + | ||
| 260 | +```bash | ||
| 261 | +# 显式预置 9.1.0 分支的闭源包；须与后续联编时 build.sh --bundle_branch=9.1.0 保持一致 | ||
| 262 | +python cmake/download_libs.py --bundle_branch=9.1.0 | ||
| 263 | +``` | ||
| 264 | + | ||
| 265 | +> 注意：离线预置的 bundle 分支须与目标环境联编时解析出的分支一致，否则联编阶段会因本地缺对应分支的包而失败。个人分支探测可能不准，建议下载与联编都显式指定 `--bundle_branch`。 | ||
| 266 | + | ||
| 258 | ### 上传依赖包 | 267 | ### 上传依赖包 |
| 259 | 268 | ||
| 260 | 在编译环境中新建一个`third_party_path`目录来存放第三方开源软件和闭源软件 | 269 | 在编译环境中新建一个`third_party_path`目录来存放第三方开源软件和闭源软件 |
| @@ -277,7 +286,9 @@ mkdir -p ${third_party_path} | |||
| 277 | |mockcpp-patch|2.7-h2|[mockcpp-2.7_py3.patch](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7_py3.patch)| | 286 | |mockcpp-patch|2.7-h2|[mockcpp-2.7_py3.patch](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7_py3.patch)| |
| 278 | |mockcpp|2.7-h2|[mockcpp-2.7.tar.gz](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7.tar.gz)| | 287 | |mockcpp|2.7-h2|[mockcpp-2.7.tar.gz](https://gitcode.com/cann-src-third-party/mockcpp/releases/download/v2.7-h2/mockcpp-2.7.tar.gz)| |
| 279 | 288 | ||
| 280 | -| 闭源二进制 | 版本 | 下载地址 | | 289 | +闭源二进制包按分支拉取，下表以默认的 `master` 分支为例；预置其它分支时把地址中的 `master` 替换为对应分支名（如 `9.1.0`），或直接用 `python cmake/download_libs.py --bundle_branch=<分支>` 自动下载对应分支的包。 |
| 290 | + | ||
| 291 | +| 闭源二进制 | 分支 | 下载地址 | | ||
| 281 | |---|---|---| | 292 | |---|---|---| |
| 282 | |cann-oam-tools-release-x86_64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-x86_64.tar.gz)| | 293 | |cann-oam-tools-release-x86_64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-x86_64.tar.gz)| |
| 283 | |cann-oam-tools-release-aarch64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-aarch64.tar.gz)| | 294 | |cann-oam-tools-release-aarch64.tar.gz|master|[Download](https://cann-3rd.obs.cn-north-4.myhuaweicloud.com/cann/oam-tools-diag/master/cann-oam-tools-release-aarch64.tar.gz)| |
| @@ -0,0 +1,361 @@ | |||
| 1 | +#!/usr/bin/env python3 | ||
| 2 | +# -*- coding: utf-8 -*- | ||
| 3 | +# ---------------------------------------------------------------------------- | ||
| 4 | +# Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 5 | +# | ||
| 6 | +# Licensed under the Apache License, Version 2.0 (the "License"); | ||
| 7 | +# you may not use this file except in compliance with the License. | ||
| 8 | +# You may obtain a copy of the License at | ||
| 9 | +# | ||
| 10 | +# http://www.apache.org/licenses/LICENSE-2.0 | ||
| 11 | +# | ||
| 12 | +# Unless required by applicable law or agreed to in writing, software | ||
| 13 | +# distributed under the License is distributed on an "AS IS" BASIS, | ||
| 14 | +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| 15 | +# See the License for the specific language governing permissions and | ||
| 16 | +# limitations under the License. | ||
| 17 | +# ---------------------------------------------------------------------------- | ||
| 18 | + | ||
| 19 | +"""闭源包(bundle)分支解析的测试：build.sh 透传、cmake 三级决策、离线预置校验。 | ||
| 20 | + | ||
| 21 | +cmake 部分不做字符串匹配，而是真正跑 cmake 求值 oam_resolve_bundle_branch()， | ||
| 22 | +这样 ahead-count 探测、白名单校验等逻辑的实际行为才被覆盖。 | ||
| 23 | +""" | ||
| 24 | + | ||
| 25 | +import re | ||
| 26 | +import shutil | ||
| 27 | +import subprocess | ||
| 28 | +import textwrap | ||
| 29 | +from pathlib import Path | ||
| 30 | + | ||
| 31 | +import pytest | ||
| 32 | + | ||
| 33 | + | ||
| 34 | +REPO_ROOT = Path(__file__).resolve().parents[5] | ||
| 35 | +INSTALL_BUNDLE_CMAKE = REPO_ROOT / "cmake" / "install_bundle.cmake" | ||
| 36 | + | ||
| 37 | +GIT = shutil.which("git") | ||
| 38 | +CMAKE = shutil.which("cmake") | ||
| 39 | + | ||
| 40 | + | ||
| 41 | +def get_build_script_content(): | ||
| 42 | + return (REPO_ROOT / "build.sh").read_text(encoding="utf-8") | ||
| 43 | + | ||
| 44 | + | ||
| 45 | +# ---------------------------------------------------------------- build.sh 透传 | ||
| 46 | + | ||
| 47 | +def test_bundle_branch_passed_without_quotes(): | ||
| 48 | + # CMAKE_ARGS 在 cmake_generate_make 里是 cmake ${cmake_args} .. 非引号展开， | ||
| 49 | + # 若给取值套引号，引号会作为字面量进入 CMake，白名单校验必然失配。 | ||
| 50 | + build_content = get_build_script_content() | ||
| 51 | + assert "-DOAM_BUNDLE_BRANCH=${BUNDLE_BRANCH}" in build_content, \ | ||
| 52 | + "应以不加引号的形式透传 OAM_BUNDLE_BRANCH" | ||
| 53 | + assert '-DOAM_BUNDLE_BRANCH=\\"${BUNDLE_BRANCH}\\"' not in build_content, \ | ||
| 54 | + "不应把转义引号拼入 CMake 参数（引号会成为字面量）" | ||
| 55 | + | ||
| 56 | + | ||
| 57 | +def test_bundle_branch_only_passed_when_explicit(): | ||
| 58 | + # 不指定 --bundle_branch 时不得透传，否则会盖掉 cmake 配置期的 git 探测。 | ||
| 59 | + build_content = get_build_script_content() | ||
| 60 | + match = re.search( | ||
| 61 | + r'if \[ -n "\$\{BUNDLE_BRANCH\}" \]; then\n(?P<body>.*?)\n fi', | ||
| 62 | + build_content, re.S) | ||
| 63 | + assert match is not None, "OAM_BUNDLE_BRANCH 应仅在 BUNDLE_BRANCH 非空时透传" | ||
| 64 | + assert "OAM_BUNDLE_BRANCH" in match.group("body") | ||
| 65 | + | ||
| 66 | + | ||
| 67 | +def test_bundle_branch_initialized_in_checkopts(): | ||
| 68 | + # 环境里的同名变量不得被当作"用户显式指定"：否则既盖掉配置期 git 探测， | ||
| 69 | + # 又绕过 --bundle_branch 解析处的字符校验（透传未校验的取值）。 | ||
| 70 | + build_content = get_build_script_content() | ||
| 71 | + checkopts = re.search(r"^checkopts\(\) \{\n(?P<body>.*?)^\}", | ||
| 72 | + build_content, re.S | re.M) | ||
| 73 | + assert checkopts is not None, "build.sh 应有 checkopts() 函数" | ||
| 74 | + init_part = checkopts.group("body").split("parsed_args=", 1)[0] | ||
| 75 | + assert 'BUNDLE_BRANCH=""' in init_part, \ | ||
| 76 | + "BUNDLE_BRANCH 应在 checkopts 解析选项前显式清空，避免继承环境变量" | ||
| 77 | + | ||
| 78 | + | ||
| 79 | +def test_bundle_branch_value_is_validated(): | ||
| 80 | + # 取值未加引号，故必须在解析处拒绝含空格/shell 元字符的分支名。 | ||
| 81 | + build_content = get_build_script_content() | ||
| 82 | + assert "^[A-Za-z0-9._/-]+$" in build_content, \ | ||
| 83 | + "--bundle_branch 取值应做字符白名单校验" | ||
| 84 | + | ||
| 85 | + | ||
| 86 | + | ||
| 87 | + ("master", True), | ||
| 88 | + ("9.1.0", True), | ||
| 89 | + ("release/1.0", True), | ||
| 90 | + ("bad name", False), | ||
| 91 | + ('x";rm -rf /', False), | ||
| 92 | + ("", False), | ||
| 93 | +]) | ||
| 94 | +def test_bundle_branch_validation_regex_behavior(value, accepted): | ||
| 95 | + # 直接验证正则本身的接受/拒绝行为，避免只断言"字面量存在"。 | ||
| 96 | + assert bool(re.fullmatch(r"[A-Za-z0-9._/-]+", value)) is accepted | ||
| 97 | + | ||
| 98 | + | ||
| 99 | +# ------------------------------------------------- cmake 分支解析（真实求值） | ||
| 100 | + | ||
| 101 | +def _run_resolve(tmp_path, *, explicit=None, refs=(), with_git=True): | ||
| 102 | + """在临时 git 仓里真正调用 oam_resolve_bundle_branch()，返回解析出的分支。 | ||
| 103 | + | ||
| 104 | + refs: 要创建的分支名序列（模拟 origin/master、origin/9.1.0-beta.2 是否存在）。 | ||
| 105 | + """ | ||
| 106 | + repo = tmp_path / "repo" | ||
| 107 | + repo.mkdir() | ||
| 108 | + subprocess.run([GIT, "init", "-q"], cwd=repo, check=True) | ||
| 109 | + subprocess.run([GIT, "config", "user.email", "t@t"], cwd=repo, check=True) | ||
| 110 | + subprocess.run([GIT, "config", "user.name", "t"], cwd=repo, check=True) | ||
| 111 | + (repo / "f").write_text("1", encoding="utf-8") | ||
| 112 | + subprocess.run([GIT, "add", "."], cwd=repo, check=True) | ||
| 113 | + subprocess.run([GIT, "commit", "-qm", "base"], cwd=repo, check=True) | ||
| 114 | + for ref in refs: | ||
| 115 | + # 用 update-ref 直接造出 refs/remotes/origin/<name>，模拟已 fetch 的远端分支。 | ||
| 116 | + subprocess.run([GIT, "update-ref", f"refs/remotes/{ref}", "HEAD"], | ||
| 117 | + cwd=repo, check=True) | ||
| 118 | + | ||
| 119 | + # install_bundle.cmake 末尾会调用 oam_install_bundle()（会联网下载）， | ||
| 120 | + # 这里只取函数定义部分，单独求值分支解析逻辑。 | ||
| 121 | + content = INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8") | ||
| 122 | + content = content.replace("oam_install_bundle()\n", "") | ||
| 123 | + | ||
| 124 | + proj = tmp_path / "proj" | ||
| 125 | + proj.mkdir() | ||
| 126 | + (proj / "bundle_funcs.cmake").write_text(content, encoding="utf-8") | ||
| 127 | + (proj / "CMakeLists.txt").write_text(textwrap.dedent(f""" | ||
| 128 | + cmake_minimum_required(VERSION 3.14) | ||
| 129 | + project(bundle_branch_probe NONE) | ||
| 130 | + set(OAM_TOOLS_DIR "{repo.as_posix()}") | ||
| 131 | + include(${{CMAKE_CURRENT_SOURCE_DIR}}/bundle_funcs.cmake) | ||
| 132 | + oam_resolve_bundle_branch(_resolved) | ||
| 133 | + message(STATUS "RESOLVED=${{_resolved}}") | ||
| 134 | + """), encoding="utf-8") | ||
| 135 | + | ||
| 136 | + args = [CMAKE, "-S", str(proj), "-B", str(tmp_path / "build")] | ||
| 137 | + if explicit is not None: | ||
| 138 | + args.append(f"-DOAM_BUNDLE_BRANCH={explicit}") | ||
| 139 | + if not with_git: | ||
| 140 | + # 强制 git 不可用，验证兜底路径。 | ||
| 141 | + args.append("-DGIT_EXECUTABLE=") | ||
| 142 | + result = subprocess.run(args, capture_output=True, text=True, check=False) | ||
| 143 | + match = re.search(r"RESOLVED=(\S*)", result.stdout) | ||
| 144 | + assert match, f"cmake 未输出解析结果:\n{result.stdout}\n{result.stderr}" | ||
| 145 | + return match.group(1) | ||
| 146 | + | ||
| 147 | + | ||
| 148 | + | ||
| 149 | +def test_resolve_explicit_takes_precedence(tmp_path): | ||
| 150 | + # 显式指定优先于 git 探测：即便 origin/master 存在也应取显式值。 | ||
| 151 | + assert _run_resolve(tmp_path, explicit="9.1.0", refs=("origin/master",)) == "9.1.0" | ||
| 152 | + | ||
| 153 | + | ||
| 154 | + | ||
| 155 | + | ||
| 156 | + "9.1.0", | ||
| 157 | + "9.1.0-beta.1", | ||
| 158 | + "9.1.0-beta.2", | ||
| 159 | + "9.1.0-beta.3", | ||
| 160 | +]) | ||
| 161 | +def test_resolve_detects_910_line(tmp_path, ref_name): | ||
| 162 | + # 同一发布线并存 9.1.0 与 9.1.0-beta.N 多个分支，任一存在都应归一到 OBS 名 9.1.0。 | ||
| 163 | + # 早期实现只硬编码了 origin/9.1.0-beta.2，其余分支会探测不到而静默回退 master。 | ||
| 164 | + assert _run_resolve(tmp_path, refs=(f"origin/{ref_name}",)) == "9.1.0" | ||
| 165 | + | ||
| 166 | + | ||
| 167 | + | ||
| 168 | +def test_resolve_ignores_branch_without_obs_package(tmp_path): | ||
| 169 | + # 9.2.0 线 OBS 上还没包，不应被探测出来（否则必然触发白名单 FATAL_ERROR）。 | ||
| 170 | + assert _run_resolve(tmp_path, refs=("origin/9.2.0-beta.1",)) == "master" | ||
| 171 | + | ||
| 172 | + | ||
| 173 | + | ||
| 174 | +def test_resolve_detects_release_line_on_non_origin_remote(tmp_path): | ||
| 175 | + # fork 场景下发布线常只在 upstream 上，不应因 remote 名不是 origin 而漏掉。 | ||
| 176 | + assert _run_resolve(tmp_path, refs=("upstream/9.1.0-beta.3",)) == "9.1.0" | ||
| 177 | + | ||
| 178 | + | ||
| 179 | + | ||
| 180 | +def test_resolve_detects_master(tmp_path): | ||
| 181 | + assert _run_resolve(tmp_path, refs=("origin/master",)) == "master" | ||
| 182 | + | ||
| 183 | + | ||
| 184 | + | ||
| 185 | +def test_resolve_falls_back_when_no_ref(tmp_path): | ||
| 186 | + # 候选 ref 都不存在（如浅克隆/未 fetch release 线）时兜底 master。 | ||
| 187 | + assert _run_resolve(tmp_path, refs=()) == "master" | ||
| 188 | + | ||
| 189 | + | ||
| 190 | + | ||
| 191 | +def test_resolve_falls_back_without_git(tmp_path): | ||
| 192 | + # git 不可用时跳过探测、兜底 master，保持无 git 环境的原有行为。 | ||
| 193 | + assert _run_resolve(tmp_path, refs=("origin/9.1.0-beta.2",), with_git=False) == "master" | ||
| 194 | + | ||
| 195 | + | ||
| 196 | +# ------------------------------------- 已有 bundle 目录的分支校验（真实求值） | ||
| 197 | + | ||
| 198 | +def _run_install_with_existing_bundle(tmp_path, *, present_branch, explicit): | ||
| 199 | + """预置一个非空 bundle 目录后真正跑 oam_install_bundle()，返回 cmake 结果。 | ||
| 200 | + | ||
| 201 | + 非空 bundle 会命中"跳过下载"分支并直接 return，故不会联网。 | ||
| 202 | + present_branch=None 模拟本改动之前拉下的、没有元数据的旧 bundle。 | ||
| 203 | + """ | ||
| 204 | + repo = tmp_path / "repo" | ||
| 205 | + bundle = repo / "bundle" | ||
| 206 | + bundle.mkdir(parents=True) | ||
| 207 | + (bundle / "aml").mkdir() | ||
| 208 | + if present_branch is not None: | ||
| 209 | + (bundle / ".bundle_branch").write_text(f"{present_branch}\n", encoding="utf-8") | ||
| 210 | + | ||
| 211 | + proj = tmp_path / "proj" | ||
| 212 | + proj.mkdir() | ||
| 213 | + (proj / "bundle_funcs.cmake").write_text( | ||
| 214 | + INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8"), encoding="utf-8") | ||
| 215 | + (proj / "CMakeLists.txt").write_text(textwrap.dedent(f""" | ||
| 216 | + cmake_minimum_required(VERSION 3.14) | ||
| 217 | + project(bundle_reuse_probe NONE) | ||
| 218 | + set(OAM_TOOLS_DIR "{repo.as_posix()}") | ||
| 219 | + include(${{CMAKE_CURRENT_SOURCE_DIR}}/bundle_funcs.cmake) | ||
| 220 | + """), encoding="utf-8") | ||
| 221 | + | ||
| 222 | + return subprocess.run( | ||
| 223 | + [CMAKE, "-S", str(proj), "-B", str(tmp_path / "build"), | ||
| 224 | + f"-DOAM_BUNDLE_BRANCH={explicit}"], | ||
| 225 | + capture_output=True, text=True, check=False) | ||
| 226 | + | ||
| 227 | + | ||
| 228 | + | ||
| 229 | +def test_existing_bundle_matching_branch_is_reused(tmp_path): | ||
| 230 | + # 分支一致时照旧复用，不应因新增校验而误报。 | ||
| 231 | + result = _run_install_with_existing_bundle( | ||
| 232 | + tmp_path, present_branch="9.1.0", explicit="9.1.0") | ||
| 233 | + assert result.returncode == 0, f"{result.stdout}\n{result.stderr}" | ||
| 234 | + assert "skip download" in result.stdout | ||
| 235 | + | ||
| 236 | + | ||
| 237 | + | ||
| 238 | +def test_existing_bundle_wrong_branch_is_rejected(tmp_path): | ||
| 239 | + # 从 master 构建后切到 9.1.0 直接 build.sh：旧 bundle 必须被拒而非静默复用。 | ||
| 240 | + result = _run_install_with_existing_bundle( | ||
| 241 | + tmp_path, present_branch="master", explicit="9.1.0") | ||
| 242 | + assert result.returncode != 0, "分支不一致的已有 bundle 应在配置阶段报错" | ||
| 243 | + combined = result.stdout + result.stderr | ||
| 244 | + assert "existing bundle" in combined and "master" in combined | ||
| 245 | + assert "--make_clean" in combined, "报错应提示如何刷新 bundle" | ||
| 246 | + | ||
| 247 | + | ||
| 248 | + | ||
| 249 | +def test_existing_bundle_without_metadata_warns(tmp_path): | ||
| 250 | + # 本改动之前拉下的 bundle 没有元数据：告警但不阻断， | ||
| 251 | + # 否则既有工作目录都得先 --make_clean 才能继续构建。 | ||
| 252 | + result = _run_install_with_existing_bundle( | ||
| 253 | + tmp_path, present_branch=None, explicit="9.1.0") | ||
| 254 | + assert result.returncode == 0, f"{result.stdout}\n{result.stderr}" | ||
| 255 | + assert "no .bundle_branch" in (result.stdout + result.stderr) | ||
| 256 | + | ||
| 257 | + | ||
| 258 | +# ------------------------------------------------------- 白名单与离线包校验 | ||
| 259 | + | ||
| 260 | +def test_known_branches_whitelist_declared(): | ||
| 261 | + # 白名单是防止拼出 OBS 上不存在的地址、下载到 403 空包的硬门禁。 | ||
| 262 | + content = INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8") | ||
| 263 | + assert 'set(OAM_BUNDLE_KNOWN_BRANCHES "master" "9.1.0")' in content | ||
| 264 | + assert "IN_LIST OAM_BUNDLE_KNOWN_BRANCHES" in content, "应对解析结果做白名单校验" | ||
| 265 | + assert "FATAL_ERROR" in content, "白名单不命中应在配置阶段直接报错" | ||
| 266 | + | ||
| 267 | + | ||
| 268 | +def test_whitelist_hint_has_no_shell_metachars(): | ||
| 269 | + # 提示语会被用户整行复制到终端，含 <> 会被 shell 当重定向解析。 | ||
| 270 | + # 只检查 FATAL_ERROR 消息正文与拼装 _hint 的语句：注释里为说明"为何不用尖括号" | ||
| 271 | + # 会出现 --bundle_branch=<name> 字样，那是文档而非用户可见提示，不应误判。 | ||
| 272 | + content = INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8") | ||
| 273 | + code_lines = [ | ||
| 274 | + line for line in content.splitlines() | ||
| 275 | + if not line.lstrip().startswith("#") | ||
| 276 | + ] | ||
| 277 | + hint_lines = [line for line in code_lines if "--bundle_branch=" in line] | ||
| 278 | + assert hint_lines, "错误提示应给出 --bundle_branch 用法" | ||
| 279 | + for line in hint_lines: | ||
| 280 | + assert "--bundle_branch=<" not in line, \ | ||
| 281 | + f"提示不应出现 --bundle_branch=<...>（尖括号是 shell 重定向元字符）: {line}" | ||
| 282 | + assert "--bundle_branch=${_b}" in content, "应逐个列出可用取值" | ||
| 283 | + | ||
| 284 | + | ||
| 285 | +def test_local_tarball_branch_is_verified(): | ||
| 286 | + # 预置包名不含分支，命中后必须校验旁写的 .branch 元数据， | ||
| 287 | + # 否则离线构建会静默混入其它分支的闭源包。 | ||
| 288 | + content = INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8") | ||
| 289 | + assert '${_local_tar}.branch' in content, "命中本地预置包时应读取 .branch 元数据" | ||
| 290 | + assert "local bundle tarball branch mismatch" in content, \ | ||
| 291 | + "预置包分支与目标分支不一致时应报错" | ||
| 292 | + | ||
| 293 | + | ||
| 294 | +def test_download_libs_writes_branch_metadata(): | ||
| 295 | + # 元数据由离线预置脚本生成，两端须配对，否则校验永远走"无元数据"告警分支。 | ||
| 296 | + content = (REPO_ROOT / "cmake" / "download_libs.py").read_text(encoding="utf-8") | ||
| 297 | + assert "def write_bundle_branch_metadata" in content | ||
| 298 | + assert "write_bundle_branch_metadata(bundle_branch" in content, \ | ||
| 299 | + "主流程应在下载后写入分支元数据" | ||
| 300 | + assert '.branch' in content | ||
| 301 | + | ||
| 302 | + | ||
| 303 | +def test_existing_bundle_dir_branch_is_verified(): | ||
| 304 | + # 从 master 构建后切到 9.1.0 直接 build.sh 会命中"bundle 已存在"分支， | ||
| 305 | + # 不校验就会静默复用 master 的闭源包——这正是本 PR 要修的场景。 | ||
| 306 | + content = INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8") | ||
| 307 | + skip_block = re.search( | ||
| 308 | + r"if\(_bundle_entries\)(?P<body>.*?)\n endif\(\)", content, re.S) | ||
| 309 | + assert skip_block is not None, "应有 bundle 已存在时跳过下载的分支" | ||
| 310 | + body = skip_block.group("body") | ||
| 311 | + assert "oam_resolve_bundle_branch" in body, \ | ||
| 312 | + "复用已有 bundle 前应先解析目标分支" | ||
| 313 | + assert "existing bundle at" in body and "FATAL_ERROR" in body, \ | ||
| 314 | + "已有 bundle 分支与目标分支不一致时应在配置阶段报错" | ||
| 315 | + # 元数据须在取包成功后写入，否则下次复用永远无从核对。 | ||
| 316 | + assert 'file(WRITE "${_bundle_meta}"' in content, \ | ||
| 317 | + "取包成功后应写入 bundle 分支元数据" | ||
| 318 | + | ||
| 319 | + | ||
| 320 | +def test_download_libs_metadata_only_for_fetched_tars(): | ||
| 321 | + # 仅按文件是否存在来写元数据，会把目录里残留的旧分支 tar 误标成本轮分支， | ||
| 322 | + # 联编时反而错误通过校验（比没有元数据更糟）。 | ||
| 323 | + content = (REPO_ROOT / "cmake" / "download_libs.py").read_text(encoding="utf-8") | ||
| 324 | + assert "downloaded = download_files_native(" in content, \ | ||
| 325 | + "download_files_native 应返回本轮成功下载的集合" | ||
| 326 | + assert "fetched_bundle_tars" in content, "应只给本轮下载成功的 bundle tar 写元数据" | ||
| 327 | + assert "write_bundle_branch_metadata(bundle_branch, fetched_bundle_tars)" in content | ||
| 328 | + # 本轮未取到的包若留着旧元数据，会被联编当成"已核对"错误放行。 | ||
| 329 | + assert "stale_meta" in content, "本轮未下载到的包应清掉其陈旧元数据" | ||
| 330 | + | ||
| 331 | + | ||
| 332 | +def test_download_libs_exits_when_no_bundle_fetched(): | ||
| 333 | + # bundle 一个都没取到时，离线预置结果不可用，应显式失败而非静默成功。 | ||
| 334 | + content = (REPO_ROOT / "cmake" / "download_libs.py").read_text(encoding="utf-8") | ||
| 335 | + assert "if not fetched_bundle_tars:" in content | ||
| 336 | + assert "sys.exit(1)" in content, "无任何 bundle 包下载成功时应以非零码退出" | ||
| 337 | + | ||
| 338 | + | ||
| 339 | +def test_download_single_file_removes_partial_file_on_failure(): | ||
| 340 | + # wget -O 失败会残留半截文件，留着会被后续当作可用预置包。 | ||
| 341 | + content = (REPO_ROOT / "cmake" / "download_libs.py").read_text(encoding="utf-8") | ||
| 342 | + wget_block = re.search( | ||
| 343 | + r'if result\.returncode != 0:(?P<body>.*?)raise RuntimeError\(f"wget download failed', | ||
| 344 | + content, re.S) | ||
| 345 | + assert wget_block is not None | ||
| 346 | + assert "os.remove(file_path)" in wget_block.group("body"), \ | ||
| 347 | + "wget 失败应删除残留的半截文件" | ||
| 348 | + | ||
| 349 | + | ||
| 350 | +def test_download_libs_whitelist_matches_cmake(): | ||
| 351 | + # 两处白名单必须同步，否则离线预置能拼出 cmake 会拒绝的分支。 | ||
| 352 | + py_content = (REPO_ROOT / "cmake" / "download_libs.py").read_text(encoding="utf-8") | ||
| 353 | + cmake_content = INSTALL_BUNDLE_CMAKE.read_text(encoding="utf-8") | ||
| 354 | + py_match = re.search(r"OAM_BUNDLE_KNOWN_BRANCHES = \((?P<v>[^)]*)\)", py_content) | ||
| 355 | + assert py_match, "download_libs.py 应声明 OAM_BUNDLE_KNOWN_BRANCHES" | ||
| 356 | + py_branches = set(re.findall(r'"([^"]+)"', py_match.group("v"))) | ||
| 357 | + cmake_match = re.search(r"set\(OAM_BUNDLE_KNOWN_BRANCHES (?P<v>[^)]*)\)", cmake_content) | ||
| 358 | + assert cmake_match | ||
| 359 | + cmake_branches = set(re.findall(r'"([^"]+)"', cmake_match.group("v"))) | ||
| 360 | + assert py_branches == cmake_branches, \ | ||
| 361 | + f"白名单不同步: py={py_branches} cmake={cmake_branches}" | ||
级别：提示 问题：BUNDLE_BRANCH 变量在拼入 CMAKE_ARGS 时未加引号：-DOAM_BUNDLE_BRANCH=${BUNDLE_BRANCH}。 影响：若分支名含空格或特殊字符，CMake 参数解析可能异常。虽然正常分支名不含空格，但缺乏防御。 修复建议：改为 -DOAM_BUNDLE_BRANCH="${BUNDLE_BRANCH}" 或在传入前校验分支名合法性。