已关闭

[RFC]: add MatmulAdd operators for fused wgrad accumulation #80

Liz创建于 27 天前关闭于 26 天前

27 天前创建

状态（Status）： Draft
作者（Authors）： @Liz_
创建日期（Created）： 2026-09-22
更新日期（Updated）： 2026-09-22
相关 Issue/PR： #182

1. 概述

1.1 简介

本提案将融合权重梯度（wgrad）累加算子 MatmulAdd（npu_matmul_add_fp32 / npu_matmul_add_fp16，ATB 后端类别）从 MindSpeed 0.16 单体仓迁移合入分层架构下的 MindSpeed-Ops 仓，并在 MegatronAdaptor（MA） 以非侵入方式补齐 Megatron-LM fused_weight_gradient_mlp_cuda.wgrad_gemm_accum_fp32/fp16 的 NPU 实现（缺省创建虚拟模块、无条件委托 MindSpeed-Ops 算子）。核心价值：在昇腾上以单 kernel 融合完成 main_grad += grad_output^T @ total_input（GEMM+累加），消除非融合路径的独立 add kernel 与 [N,K] 中间缓存；在分布式优化器/fp32 梯度累加等主流大模型训练配置下，MLP 形状 wgrad 实测提速约 15%–17%；精度、性能、内存三类 ATK 测试全部 Pass，且不修改 Megatron-LM 与 MindSpeed 上游任何源码。

1.2 动机

使用场景：Megatron 风格大模型训练中，每个 TP 线性层（attention QKV/输出投影、MLP fc1/fc2、output_layer 词表并行、MoE 专家/路由线性层）在每个 micro-batch 反向都要执行一次 wgrad 融合累加（gradient_accumulation_fusion=True 时），是训练反向的高频热点算子。

当前痛点：

NPU 上不存在 CUDA 扩展 fused_weight_gradient_mlp_cuda，018 分层栈（Megatron-LM + MA + TransformerEngineNPU + 可插 MindSpeed）此前依赖 TENPU 内部路径，Megatron 原生 TP 线性层的 wgrad 缺少与 016 单体版对齐的融合实现；
016 已验证的 ATB 融合 MatmulAdd 算子耦合在 MindSpeed 单体仓内，018 分层架构无法复用，算子资产（文档/UT/ATK 交付件）也不符合 MindSpeed-Ops 仓的合入规范（docs/zh/ops.md）；
非融合路径（matmul + add_ 两步分解）多一次 kernel 启动、多读写一份 [N,K] 全局缓存，且中间结果多一次半精度舍入；016 vs 018 的性能对比排查正是由该差异触发。

必要性与用户价值：分布式优化器 / --accumulate-allreduce-grads-in-fp32 配置下 main_grad 为 fp32，命中 fp32 API → ATB 融合 kernel 生效（实测收益见 §3.3）；当前 bf16 main_grad 配置亦受益于 fp16 API 的单 addmm_ 实现（较 016 两步分解快 2%–3%、少一次舍入）。算子按 ops.md 全套交付（8 节文档、双标杆 UT、ATK 三任务），后续可独立验收与演进。

不做的影响：018 分层栈在 fp32 main_grad 配置下持续损失 ATB 融合收益，大形状 wgrad 落后 016 单体版；算子继续滞留单体仓，无法按新仓规范维护、测试与复用。

1.3 目标

目标：

MatmulAdd 按 ops_dev_guide / ops.md 合入 MindSpeed-Ops：新增 api/atb 类别（Python 入口 + op_builder JIT 基础设施 + csrc ATB C++ 源 + 8 节文档 + 算子清单/简介更新）；
MA 以"虚拟模块缺省创建 + 无条件委托"方式补齐 wgrad_gemm_accum_fp32/fp16，门控完全由 Megatron 原生参数（gradient_accumulation_fusion）与 main_grad.dtype 决定；
依赖缺失/运行失败显式报错并给出可操作指引（安装 MindSpeed-Ops 或 --no-gradient-accumulation-fusion），不做静默降级；
交付精度（双标杆 UT + ATK accuracy）、性能（实测基准 + ATK performance）、内存（ATK memory）全套证据，全部 Pass。

非目标（边界）：

不修改 016/018 的 MindSpeed 原文件、Megatron-LM 上游、TENPU 组件源码；
不 patch TENPU 内部 matmul_add（TE 路径 wgrad 仍由 TENPU 承担；是否覆盖属 phase-2，见 §7）；
不改变训练脚本的精度配置选择（是否开启 fp32 梯度累加由用户决定）；
不支持在非昇腾硬件执行 ATB 路径（仓内仅保留可移植参考实现，见 §3.4）；
MA 不新增自有后端切换开关（初版 MA_WGRAD_IMPL 三态方案已被替代，见 §3.2）。

2. 用例分析

用例	场景	功能点	关键指标/要求
UC1 fp32 main_grad 训练	分布式优化器 / `--accumulate-allreduce-grads-in-fp32`，Megatron 原生 TP 线性层 wgrad	`wgrad_gemm_accum_fp32` → `npu_matmul_add_fp32` → A2/A3 ATB 融合 kernel	MLP 形状（fc1 [24576,6144]、fc2 [6144,24576]）wgrad 提速约 15%–17%；词表形状不劣化；精度满足 single_bm 高精度标准
UC2 bf16/fp16 main_grad 训练	当前 dsa_fused 脚本配置（`--bf16` 无 fp32 累加）	`wgrad_gemm_accum_fp16` → `npu_matmul_add_fp16`（单 `addmm_`）	较 016 两步分解快 2%–3%；单次舍入，精度不劣于分解
UC3 依赖缺失环境	未安装 MindSpeed-Ops / 无 atb api / ATB 运行失败（如缺 cann-nnal）	显式 `RuntimeError`，报错含安装指引与 `--no-gradient-accumulation-fusion` 关闭方式	不静默降级、不吞错（MA 红线"不兼容要拦截"）
UC4 A5 芯片	Ascend910_95/950	fp32 API 自动走 `addmm_`（芯片版本探测分流）	无 ATB 依赖亦可运行
UC5 竞品/非 NPU 环境	精度对比、竞品测试	ATB 路径为昇腾亲和接口，不可在竞品执行；仓内保留可移植 torch 实现（`addmm_` 语义），ATK 插件 cpu/gpu 分支即该实现	UT 条件跳过（ops.md L34 允许）；竞品可复用同一套 ATK 用例

DFX 要求：兼容性（虚拟模块符号在 MA 支持的 Mcore 分支范围内缺省创建；芯片 A2/A3/A5 分流）、可维护性（算子资产归 MindSpeed-Ops、框架适配归 MA，分层内聚；csrc 与 016 语义一致仅格式化）、可测试性（双标杆 UT + 接口级 MA 单测 + ATK 三任务全可复现）、可靠性（zero-shape 守卫、原地累加语义显式文档化、错误原样传播、JIT 产物缓存）。

使用限制与约束：输入为 2D 张量（total_input [M,K]、grad_output [M,N]、grad [N,K]）；dtype 组合固定（操作数 bf16/fp16 且二者一致；grad fp32 或与操作数同 dtype）；grad 原地累加（调用方持有 main_grad 生命周期）；ATB 路径依赖 cann-nnal（ATB_HOME_PATH）与本机 C++ 工具链（首次 JIT 约 30s，产物缓存）。

3. 方案设计

3.1 总体方案

调用链与架构（自顶向下）：

Megatron-LM  tensor_parallel/layers.py（上游，不修改）
  └─ gradient_accumulation_fusion=True 时按 weight.main_grad.dtype 路由
       fp32 → fused_weight_gradient_mlp_cuda.wgrad_gemm_accum_fp32(total_input, grad_output_2d, main_grad)
       fp16/bf16 → …wgrad_gemm_accum_fp16(…)
            │  （NPU 无此 CUDA 扩展：MA requirements 以"虚拟模块缺省创建"注册同名符号，lazy import）
            ▼
MegatronAdaptor  patches/transformer_engine/transformer_engine.py
  npu_wgrad_gemm_accum_fp32/fp16(total_input, grad_output, main_grad)
    └─ 无条件委托（仅做依赖缺失的显式报错包装，无自有开关）
            ▼
MindSpeed-Ops  mindspeed_ops/api/atb/npu_matmul_add.py
  npu_matmul_add_fp32:  zero-shape 守卫 → 芯片探测（lru_cache）
        A5        → grad.addmm_(grad_output.t(), total_input)
        A2/A3/其他 → op_builder.MatmulAddOpBuilder().load()   # JIT，产物缓存
                     └→ ATB 融合 MatmulAdd kernel（C++：csrc/atb/matmul_add.cpp + atb_adapter）
  npu_matmul_add_fp16:  zero-shape 守卫 → grad.addmm_(grad_output.t(), total_input)  # 单 kernel
            ▼
CANN 9.0.0 + nnal ATB（ATB_HOME_PATH；链接 -lasdops -llcal -latb）

核心逻辑：门控完全由 Megatron 原生决定（是否融合看 gradient_accumulation_fusion，走哪个变体看 main_grad.dtype）；MA 层只做符号补齐与委托；MindSpeed-Ops 层做守卫、芯片分流与后端选择。fp32 API 在 A2/A3 走 ATB 融合（注意底层 C++ 入参序为 (grad_output, total_input, grad)，Python 入口保持与 Megatron 一致的 (total_input, grad_output, grad) 并在入口内换序）；A5 与 fp16 API 走可移植 addmm_。

限制与前置条件：UC1 的 15%–17% 收益需同时满足——fp32 main_grad 配置、A2/A3 芯片、cann-nnal 环境（ATB_HOME_PATH 已 source）、MindSpeed-Ops 可导入；词表并行大形状（output_layer）为打平（32.75 vs 32.27ms）；bf16 main_grad 配置命中 fp16 API，收益为 2%–3%（对两步分解）。

3.2 技术选型

候选方案	优点	缺点	结论
ATB 融合 MatmulAdd（选定，fp32 路径）	单 kernel GEMM+累加；016 已生产验证；MLP 形状实测 15%–17% 收益；免 [N,K] 中间缓存	依赖 cann-nnal 与 JIT 工具链；仅 A2/A3	✅ 选用
单 `addmm_`（选定，fp16 路径与 A5）	全芯片可用、零额外依赖、单 kernel、单次舍入	fp32 大形状下无 ATB 融合收益	✅ 选用
纯 `addmm_`（放弃 ATB）	最简、无 JIT	丢失 UC1 主要收益（实测 5.90→6.90ms，fc1）	❌
patch TENPU `matmul_add`	可覆盖 TE 路径 wgrad	TENPU 为独立交付组件，侵入修改耦合升级；缺端到端数据	❌ 暂缓，列 §7 phase-2
MA 内嵌 ATB C++/JIT 代码	少一层依赖	违背分层职责（算子资产应归 MindSpeed-Ops），不可复用	❌
从 016 MindSpeed 直接 import	零迁移成本	018 为可插拔 MindSpeed，不应为单算子引入单体依赖	❌
Triton 融合 kernel	仓内已有 triton 类别基建	wgrad 为 Cube 主导算子，昇腾 Triton 大 GEMM+累加不成熟；ATB 已有生产级实现	❌
MA 层 `MA_WGRAD_IMPL=auto/atb/tenpu` 三态开关 + 异常静默回退（初版实现，已被替代）	平滑过渡、旧环境无感	与 MA 红线冲突（门控应由训练参数判定，不新增自有开关）；静默回退掩盖环境问题；双路径扩大维护/测试面	❌ 由 `989cdf8` 重构为无条件委托 + 显式报错

3.3 功能与性能设计

功能定义：grad += grad_output^T @ total_input，grad 原地更新，无返回值。zero-shape 输入直接 no-op（空 wgrad GEMM 守卫）。

核心流程（fp32 API）：① zero-dim 守卫 → ② get_npu_version()（torch_npu.npu.get_device_name()，lru_cache；Ascend910_95/950→A5、Ascend910_93→A3、Ascend910B→A2）→ ③ A5：addmm_；否则 MatmulAddOpBuilder().load()（JIT：csrc/atb/matmul_add.cpp + utils/atb_adapter.cpp + csrc/flop_counter；ATB 头/库路径取 ATB_HOME_PATH，缺失则 RuntimeError("please install cann-nnal package first")；产物缓存于 ~/.cache/torch_extensions）→ ④ 调 ATB 融合 kernel（入参换序 (grad_output, total_input, grad)）。fp16 API：① 守卫 → ② 单 addmm_。

数据模型：无新增/变更数据结构；main_grad 由 Megatron 分配与持有，算子仅原地累加。影响范围：所有经 fused_weight_gradient_mlp_cuda.wgrad_gemm_accum_* 的 Megatron 原生 TP 线性层 wgrad（attention QKV/投影、MLP fc1/fc2、output_layer、MoE 线性层）；TE 模块内部 wgrad 路径不受影响。

性能实测（910B3/A2，M=4096，bf16 操作数，20 次取中位，同步计时；单位 ms）：

fp32 main_grad（UC1，ATB 融合 vs 单 addmm_ vs 两步分解）：

wgrad 形状 [N,K]	ATB 融合	addmm_	两步分解	ATB 收益
output_layer [129280,6144]	32.75	32.27	35.17	与 addmm_ 打平，较分解 +6.9%
fc1 [24576,6144]	5.90	6.90	6.36	较 addmm_ 约 +15%（耗时比 1.17）
fc2 [6144,24576]	6.15	7.08	6.63	较 addmm_ 约 +15%（耗时比 1.15）

bf16 main_grad（UC2，单 addmm_ vs 016 两步分解）：

wgrad 形状 [N,K]	addmm_（本提案）	两步分解（016）
output_layer [129280,6144]	33.11	33.18
fc1 [24576,6144]	5.84	6.01
fc2 [6144,24576]	6.09	6.25

ATK 任务级结果（小 shape 用例集，双 NPU 节点"融合 vs 非融合分解"对比，详见 §4）：fp32 平均 device 性能比 1.3662（小 shape 下分解的独立 add kernel 启动/访存开销占比更大）；fp16 1.0007（addmm_ 与分解打平，符合"fp16 收益集中在大 MLP 形状"的结论）；内存通过率均 100%（融合省去 [N,K] 中间张量）。

开销说明：ATB C++ 扩展首次 JIT 编译约 30s（一次性，按 Python 版本缓存）；芯片探测与 builder load 均有缓存，稳态调用无额外 Python 开销热点。

3.4 安全隐私与 DFX 设计

安全隐私：算子为纯本地计算，无网络 IO、无用户数据采集/传输；JIT 编译仅使用随包分发的仓内 C++ 源码与本机工具链；bandit 安全扫描通过（仓内 pre-commit 强制）。无隐私影响面。

DFX：

兼容性：MA 侧以"虚拟模块缺省创建"注册符号，不依赖 Mcore 内部实现细节，随 MA-Mcore 分支一一配套演进；芯片兼容 A2/A3（ATB）与 A5（addmm_ 分流），未知芯片回退 addmm_ 语义；MindSpeed-Ops 缺失时显式报错（受控 Breaking，见 §5）；竞品不可执行 ATB 路径，按 ops.md 昇腾亲和条款在仓内保留可移植实现。
可维护性：csrc 与 016 原件语义一致（去空白 md5 校验通过），仅按仓内 .clang-format 规范化；JIT 复用仓内 op_builder 基建；文档按 ops.md 8 节模板；ATK 资产按 tests/atk_tests/<backend>/<op>/ 仓内约定布局（atb 类别首例）。
可测试性：三层测试（§4）全部可在 910B3 单机复现；ATK 用例 -s 45 种子固定可复现；执行插件三角色（npu 待测/npu 分解标杆/cpu-gpu 可移植参考）支持后续回归与竞品对比。
可靠性：zero-shape 守卫防非法 GEMM；原地累加语义与 fp32 累加精度在文档"注意事项"显式说明；ATB 运行期错误原样传播（不静默换后端），报错信息含处置指引；column_parallel_linear_init_wrapper 将底层符号缺失转译为面向用户的可操作错误。
可观测性：错误路径统一给出"安装 MindSpeed-Ops / 检查 CANN / --no-gradient-accumulation-fusion"指引；性能回归可由 ATK performance 基线（--is_compare false + --bm_file）持续监控。

3.5 编程与调用设计

3.5.1 编程模型基本设计

开发环境：

要素	要求
硬件	Ascend 910B（A2，已验证）；A3/A5 逻辑支持、待实测（§7）
软件栈	CANN 9.0.0（toolkit + nnal ATB `set_env.sh`）、torch 2.7.1、torch_npu 2.7.1.post4、Python 3.10/3.12
框架	Megatron-LM（与 MA 分支配套版本）、MegatronAdaptor、MindSpeed-Ops（pip editable 或 PYTHONPATH）
工具链	本机 C++ 编译器（ATB JIT）；pytest（UT）；ATK 26.4.30（验收）

开发约束：仅 NPU 后端执行 ATB 路径；2D 张量与固定 dtype 组合（§2）；grad 原地累加、调用方保证 main_grad 生命周期与并发安全（Megatron wgrad 串行调用约定内）；ATB 路径依赖 ATB_HOME_PATH；竞品环境仅可运行可移植参考实现。

可验收设计：验收环境=上表 910B3 环境；验收标准与实测结果——MindSpeed-Ops UT 15/15 通过、MA 接口级单测 4/4 通过、ATK accuracy 8/8 Pass ×2 接口、ATK performance 均值比 ≥0.95 且判定 Pass（fp32 实测 1.3662 / fp16 1.0007）、ATK memory 比值 ≤1.1 且通过率 100% ×2 接口；验收命令见归档指南 atk-report/README.md（原仓内测试指南，按仓内惯例移出）。

3.5.2 接口定义与设计

3.5.2.1 npu_matmul_add_fp32

接口描述：fp32 权重梯度缓冲的融合 wgrad 累加：grad += grad_output^T @ total_input。A2/A3 经 JIT 调 ATB 融合 MatmulAdd kernel；A5 走 addmm_。
接口原型：mindspeed_ops.api.atb.npu_matmul_add.npu_matmul_add_fp32(total_input, grad_output, grad) -> None
输入/输出参数：

参数名称	输入/输出	类型	描述	取值范围
total_input	输入	torch.Tensor	前向输入 [M, K]	dtype bf16/fp16，与 grad_output 一致；含 0 维时整体 no-op
grad_output	输入	torch.Tensor	输出梯度 [M, N]	同上
grad	输入/输出	torch.Tensor	权重梯度缓冲 [N, K]，原地累加	dtype fp32

返回参数：无（原地更新 grad）。
异常处理：ATB 环境缺失 → RuntimeError("please install cann-nnal package first")；JIT 编译失败按 torch cpp_extension 异常传播。
约束说明：仅 2D；操作数 dtype 必须一致；fp32 中累加（bf16/fp16 乘积在 fp32 中精确）。
变更说明：新增（迁自 MindSpeed 0.16 mindspeed/ops/npu_matmul_add.py，C++/ATB 源未改动，Python 入口重新宿主化）。
调用参考代码：

from mindspeed_ops.api.atb.npu_matmul_add import npu_matmul_add_fp32
# total_input: [M, K] bf16；grad_output: [M, N] bf16；main_grad: [N, K] fp32
npu_matmul_add_fp32(total_input, grad_output, weight.main_grad)

3.5.2.2 npu_matmul_add_fp16

接口描述：半精度权重梯度缓冲的 wgrad 累加，单次融合 addmm_（fp32 中累加、一次舍入到 grad dtype）。
接口原型：mindspeed_ops.api.atb.npu_matmul_add.npu_matmul_add_fp16(total_input, grad_output, grad) -> None
输入/输出参数：同 3.5.2.1，唯 grad dtype 为 fp16/bf16 且与操作数一致。
返回参数：无。
异常处理：torch addmm_ 原生异常（dtype/形状不匹配等）。
约束说明：同 3.5.2.1；全芯片可用（无 ATB 依赖）。
变更说明：新增；与 016 源实现（matmul + add_ 两步分解）存在有意偏差——改为单 addmm_，少一次 kernel 与中间舍入，实测更快（§3.3）且精度不劣（双标杆 UT + ATK 精度 Pass 佐证）。
调用参考代码：同上，替换函数名。

3.5.2.3 npu_wgrad_gemm_accum_fp32 / npu_wgrad_gemm_accum_fp16（MA 补丁入口）

接口描述：Megatron fused_weight_gradient_mlp_cuda.wgrad_gemm_accum_fp32/fp16 的 NPU 等价实现；无条件委托对应 MindSpeed-Ops 算子，是否执行及变体选择完全由 Megatron-LM（gradient_accumulation_fusion、main_grad.dtype）决定。
接口原型：megatron_adaptor.patches.transformer_engine.transformer_engine.npu_wgrad_gemm_accum_fp32(total_input, grad_output, main_grad) -> None（fp16 同构）
输入/输出参数：与 Megatron 上游同名符号一致（drop-in）：total_input [M,K]、grad_output [M,N]、main_grad [N,K] 原地累加。
返回参数：无。
异常处理：mindspeed_ops 或其 atb api 不可导入 → RuntimeError（指引：安装 MindSpeed-Ops，或 --no-gradient-accumulation-fusion 关闭融合）；算子运行期异常（如缺 cann-nnal）原样传播，无静默后端切换。
约束说明：随 import megatron_adaptor 经 features_manager/requirements 以虚拟模块缺省创建方式注册（lazy import，符号 fused_weight_gradient_mlp_cuda.wgrad_gemm_accum_fp32/fp16）。
变更说明：989cdf8 由"TENPU 委托 + env 三态开关"重构为"MindSpeed-Ops 无条件委托"。
调用参考代码：用户无感——训练入口 import megatron_adaptor 后由 Megatron 原生调用。

上下游生态适配：签名与 Megatron-LM CUDA 扩展同名符号一致（drop-in 替换）；数值语义对齐 torch.Tensor.addmm_（beta=1, alpha=1）/ aclnn Addmm；ATB 底层入参序差异（(grad_output, total_input, grad)）封装在 MindSpeed-Ops 入口内，不外泄。

3.5.3 编程手册设计

不单独输出新手册，由既有文档体系承载并随本提案更新：

文档	角色	变更方式
MindSpeed-Ops `docs/zh/atb/matmul_add.md`	算子手册（ops.md 8 节：概述/签名/参数/原理/示例/性能对比/常见训练 Shape/注意事项）	随算子合入 PR 新增并持续维护
MindSpeed-Ops `docs/zh/supported_operators.md`、`docs/zh/introduction.md`	算子清单注册、后端类别说明（三类→四类，新增 atb）	同上
MA `docs/zh/patch_list.md`（L34-35）	补丁清单：符号、委托关系、兼容性说明与处置指引	随 MA PR 更新
MA README / FAQ 安装指引	MindSpeed-Ops 依赖的安装说明（`94e5278`）	已更新
ATK 测试指南（归档 `atk-report/README.md`）	复现命令、精度标准、交付件说明	按仓内惯例不入仓，归档保存

4. 测试设计

单元测试（unit test）：

MindSpeed-Ops tests/unit_tests/atb/test_npu_matmul_add.py，15 用例，真机 15/15 通过：
- 双标杆对齐（ops.md §3）：CPU 高精度参考（float64：base + lhs @ rhs）+ torch 小算子参考（clone().addmm_），每用例双断言；
- 覆盖矩阵：操作数 dtype {bf16, fp16} × grad dtype {fp32（fp32 API）/ 同操作数（fp16 API）} × shape {(64,32,16), (256,128,64), (512,256,1024)}（含 K=1024 归约维）；
- 专项：zero-shape 守卫、跨调用原地累加（2× 期望值）、grad dtype 校验；
- 可移植性：pytest.importorskip + requires_npu 条件跳过（非 NPU 环境不失败）；ATB 缺失（RuntimeError 含 "cann-nnal"）时相应跳过。
MA test/unit_tests/patches/transformer_engine/test_npu_wgrad_gemm_accum.py，2 类 4 用例，全部通过（接口级用例，符合 MA 测试约定）：fp32/fp16 委托正确性；MA_WGRAD_IMPL 及旧分发助手已移除且任意取值被忽略；算子异常原样传播（无静默切换）；依赖缺失显式报错。

集成/验收测试（ATK，integration）：资产 tests/atk_tests/atb/matmul_add/（用例设计 yaml ×2 + 参数约束插件 + 执行插件）；每接口 8 用例（dtype × shape 全交叉，种子 45 可复现）；三类任务结果（910B3 实测，全部 Pass）：

任务	节点组合	fp32 接口	fp16 接口
accuracy	npu（待测）vs cpu（高精度标杆，single_bm）	8/8，通过率 100.0，Pass	8/8，通过率 100.0，Pass
performance_device	npu dev0（融合）vs npu dev1（非融合分解）	平均性能比 1.3662，Pass	平均性能比 1.0007，Pass
memory_device	同上	内存通过率 100.0，Pass（≤1.1）	内存通过率 100.0，Pass（≤1.1）

交付件（PR 截图来源）：6 份 xlsx 报告 + 6 份控制台日志 + SUMMARY，归档于 /home/l00661657/version-mul/018/atk-report/；按团队流程 xlsx 另存 ATK_CIDA 仓。

端到端测试（e2e）：以 dsa_fused 训练脚本（910B3）对比"融合 wgrad 生效（安装 MindSpeed-Ops + source ATB 环境 + gradient_accumulation_fusion 开启）" vs "--no-gradient-accumulation-fusion 基线"：验证 loss 曲线逐 step 对齐与吞吐不劣化；该 A/B 同时作为 phase-2（§7）的决策输入。当前状态：环境与代码就绪，正式 A/B 数据待跑（列 §7）。

静态检查：两仓 pre-commit 全绿（ruff check/format、pylint、bandit、codespell、typos、trailing-whitespace、end-of-file-fixer、check-yaml 等；markdownlint 钩子源不可达时按其规则人工核查通过）。

5. 缺点和风险

风险/缺点	影响	应对
受控 Breaking Change：MA wgrad 委托对象由 TENPU 改为 MindSpeed-Ops，未安装 MindSpeed-Ops 的环境开启融合时显式报错	018 栈用户需新增安装 MindSpeed-Ops	报错信息含安装/关闭双指引；README/FAQ 安装指导（`94e5278`）；发布说明标注依赖
收益有前置条件：bf16 main_grad 配置命中 fp16 API（`addmm_`），ATB 融合收益仅 fp32 main_grad + A2/A3 生效	用户预期偏差（"装了没提速"）	文档"性能对比/注意事项"明确条件；FAQ 说明配置方法
ATB JIT 首次编译约 30s，且依赖本机 C++ 工具链与 cann-nnal	首跑变慢；无工具链环境报错	编译产物缓存；缺 ATB 时显式 `RuntimeError` 指引；预编译打包列 §7 开放问题
CANN/ATB 版本耦合（验证于 CANN 9.0.0）	CANN 升级潜在 ABI/接口漂移	`op_builder` 探测失败显式报错；CI 锁版本回归
fp16 实现与 016 分解存在有意偏差（单 `addmm_`）	数值差异：少一次中间半精度舍入（精度更优）；行为差异：单 kernel	双标杆 UT + ATK 精度全 Pass 佐证；文档"注意事项"显式声明偏差
auto 时代静默回退已移除：ATB 运行期错误直接失败	环境问题暴露为训练中断（设计取舍）	符合"不兼容要拦截"红线；报错含处置指引；比静默降级更利于定位
原地累加语义误用（别名/并发/autograd 交互）	数值错误	文档注意事项 7 条；UT 覆盖跨调用累加与守卫
A3/A5 未在真机验证	芯片矩阵声明超出实测范围	分流逻辑 UT/代码评审覆盖；真机验证列 §7
维护成本：新增 atb 类别（csrc 约 600 行 + builder 约 200 行 + 文档/UT/ATK 资产）、MA 侧约 60 行	长期维护投入	资产全部符合仓规范（可复用基建/模板）；csrc 与 016 同源降低认知成本
API/版本兼容	无对外 API 破坏（全为新增符号；Megatron 同名符号语义 drop-in 一致）	—
旧版本迁移	016 用户零迁移（016 未改动）；018 用户安装 MindSpeed-Ops 即得	安装指引已入文档

6. 现有技术

现有技术	借鉴	差异
MindSpeed 0.16 单体仓 `mindspeed/ops/npu_matmul_add` + `csrc/atb`（迁移源）	ATB 融合 MatmulAdd C++/kernel 实现与芯片分流思路	本提案：独立分层仓 + 复用 `op_builder` JIT 基建；fp16 由两步分解改单 `addmm_`；补齐 ops.md 全套交付件（8 节文档/双标杆 UT/ATK 三任务）；MA 侧显式报错替代静默回退
Megatron-LM / NVIDIA `fused_weight_gradient_mlp_cuda.wgrad_gemm_accum_fp32/fp16`	融合 wgrad 的接口形态与 dtype 路由（`main_grad.dtype` 决定变体）	昇腾 drop-in 等价实现；以虚拟模块缺省创建方式非侵入补齐
TransformerEngineNPU 融合 GEMM 累加（旧委托路径）	fp16 单 kernel `addmm_` 语义与其对齐	TE 内部 wgrad 仍归 TENPU；本提案覆盖 Megatron 原生 TP 线性层路径
ATB（Ascend Transformer Boost）models 融合算子库	底层 MatmulAdd kernel 与链接方式（`-lasdops -llcal -latb`）	经 JIT 适配器封装为 torch 张量接口，随包分发源码
`torch.Tensor.addmm_` / aclnn Addmm	可移植实现语义（亦为 CPU/GPU 标杆与竞品参考实现）	ATB 路径在大形状 Cube 融合上更优（§3.3）
MindSpeed-Ops 仓 triton/aclnn 类别及其 ATK 测试流程	目录约定（`tests/atk_tests/<backend>/<op>/`）、用例设计 yaml + 约束/执行插件模式、三任务验收流程	本提案为 atb 类别首例，并扩展了"双 NPU 节点融合 vs 非融合"的性能/内存对比方法（CPU 节点无 device 指标，比值不可算）
NVIDIA apex / PyTorch `addmm(beta=1)` 融合累加惯例	`C += A^T·B` 的原地累加 API 形态	与 Megatron 签名对齐而非 apex 签名

7. 未解决问题

NAN

欢迎加入社区，感谢您对社区的贡献 🎉!

xmz成员

27 天前评论：

🎉 您好，欢迎向 MindSpeed-Ops 提交 Issue！ 🎊
我们已经收到您的反馈，感谢您对开源社区的支持！💖
⏱️ 处理时效：维护团队将在 24小时内查看并回复您的问题（工作日）。
🤖 自助查询：在等待期间，您可以先使用 Agent 进行问题分析，可能可以得到规避方案。
📝 温馨提示：请确保 Issue 描述清晰，包含复现步骤和日志，这将帮助我们更快定位问题。
谢谢您的配合！🙏✨

X xmz成员

27 天前添加了label：triaged

ascend-robot成员

27 天前添加了label：rfc

L Liz

27 天前修改标题为 “[RFC]: add MatmulAdd operators for fused wgrad accumulation”，原标题为“[RFC]: ”

27 天前修改了issue 的描述

27 天前修改了issue 的描述

27 天前修改了issue 的描述

27 天前评论：

MatmulAdd ATK 交付件汇总（2026-09-22 最终版）

环境：Ascend 910B3 ×8（A2）、CANN 9.0.0 + ATB、conda lz_base（Python 3.10、torch 2.7.1、torch_npu 2.7.1.post4）、ATK 26.4.30。
待测算子：mindspeed_ops.api.atb.npu_matmul_add.npu_matmul_add_fp32/fp16（PYTHONPATH 指向本仓）。
用例：每接口 8 条 = 操作数 dtype {bf16, fp16} × shape (M,K,N) {(64,32,16),(128,128,128),(256,512,128),(512,256,1024)}，全交叉覆盖（-s 45 可复现）。
本目录 6 份报告全部由最终版执行插件 atb_matmul_add.py（npu/cpu/gpu 三角色，pre-commit 全绿）产生。

结果总览（全部 Pass）

任务	节点组合	fp32 接口	fp16 接口
accuracy	npu dev0(待测) vs cpu(高精度标杆)	8/8，通过率 100.0，Pass	8/8，通过率 100.0，Pass
performance_device	npu dev0(融合) vs npu dev1(非融合分解)	平均性能比 1.3662，通过率 100.0，Pass	平均性能比 1.0007，通过率 87.5，整体 Pass
memory_device	npu dev0(融合) vs npu dev1(非融合分解)	内存通过率 100.0，Pass（比值≤1.1）	内存通过率 100.0，Pass（比值≤1.1）

语义说明：

精度：CPU 标杆为可移植 torch 实现（fp32 中计算、一次舍入到 grad dtype），single_bm/high_precision 标准，阈值及理由见各 yaml 注释。
性能比 = 标杆耗时 / 待测耗时，越大越好（标准：均值≥0.95、单用例≥0.98）。fp32 待测为 ATB 融合 MatmulAdd kernel，对比非融合 mm+add_ 分解（MindSpeed 源码实现形态）：小 shape 融合平均快 36.6%（省独立 add kernel 的启动与访存）；大训练 shape 13–17% 收益见 docs/zh/atb/matmul_add.md 性能对比章节。
fp16 待测为单 kernel addmm_，与分解均值打平（1.0007）；收益集中在大 MLP shape（见文档）。小 shape 单用例比值有 ±2% 运行间波动：三轮实测均值 0.9931(单用例75%) / 1.0061(100%) / 1.0007(87.5%)，整体判定三轮均 Pass。
内存比 = 待测/标杆 ≤1.1 判 Pass；融合算子省去 [N,K] 中间临时张量，全部用例通过。
竞品说明：ATB 路径为昇腾亲和接口，不支持竞品执行；执行插件的 cpu/gpu 分支即仓内保留的可移植竞品实现（ops.md L25），竞品节点可复用同一套用例做精度/性能对比。

交付件路径（相对 /home/l00661657/version-mul/018/atk-report 目录）

正式报告（PR 截图来源；xlsx 含逐用例精度详情 / Device性能(us) / 内存字段，failed cases sheet 均为空）：

交付件	路径
精度 fp32	atk_output/all_matmul_add_fp32_2026-09-22-14-19-05-502423/report/all_matmul_add_fp32_reports_2026-09-22-14-19-06.xlsx
精度 fp16	atk_output/all_matmul_add_fp16_2026-09-22-14-19-30-365038/report/all_matmul_add_fp16_reports_2026-09-22-14-19-31.xlsx
性能 fp32	atk_output/all_matmul_add_fp32_2026-09-22-14-21-16-735370/report/all_matmul_add_fp32_reports_2026-09-22-14-21-18.xlsx
内存 fp32	atk_output/all_matmul_add_fp32_2026-09-22-14-22-58-241657/report/all_matmul_add_fp32_reports_2026-09-22-14-22-59.xlsx
性能 fp16	atk_output/all_matmul_add_fp16_2026-09-22-14-24-24-711434/report/all_matmul_add_fp16_reports_2026-09-22-14-24-26.xlsx
内存 fp16	atk_output/all_matmul_add_fp16_2026-09-22-14-26-04-923310/report/all_matmul_add_fp16_reports_2026-09-22-14-26-06.xlsx

控制台日志（含汇总表格，可直接截图）：atk_output/console/*.log（6 份，与上表一一对应）。
生成的用例（JSON/Excel/CSV）：result/matmul_add_fp32/、result/matmul_add_fp16/。
复现命令见本归档目录 README.md（原仓内测试指南，已按仓内惯例移出仓库）。按团队流程，xlsx 报告需另存至 ATK_CIDA 仓归档。

Liz

27 天前评论：

accuracy_fp16_npu_vs_cpu.log

tensorflow is not install in env
tensorflow is not install in env
tensorflow is not install in env
ct-tool not installed, please install it first.
[1;36m[2026-09-22 12:52:09][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mtask_output_path: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-52-09-180207[0m[0m
[1;36m[2026-09-22 12:52:09][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mstart atk task: namespace(case='result/matmul_add_fp16/json/all_matmul_add_fp16.json', task=['accuracy'], plugin_path='atb_matmul_add.py', fast_case=None, slice_input=None, slice_input_ratio=None, nodes=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', white_list=None, black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, single_process=False, mssanitizer=False, ms_log_path=None, max_task=100, trace=False, timeout=None, db_timeout=None, db_path=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, atb_api_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, bm_output_path_adr='', sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, get_tiling_key_path=None, need_time=True, ct_tool=False, gm_init_flag=False, fluctuation_check=False, clean_l2_cache=False, nodes_list=[{'backend': <NodeType.NPU: 'npu'>, 'devices': [0], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}, {'backend': <NodeType.CPU: 'cpu'>, 'name': None, 'devices': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}])[0m[0m
[1;36m[2026-09-22 12:52:09][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36matb_matmul_add.py[0m:[1;36m38[0m] [1m[34mmatmul_add_atk is already registered, repeated registration will coverage it, new module path is atb_matmul_add[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[823325][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[823325][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@create_dataset', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823327][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823329][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_0', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823331][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823334][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[823348][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[823348][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@cpu_run', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[823356][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[823356][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@compare', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks start!!!, case_config_name:matmul_add_fp16, all_cases_num: 8[0m[0m

ATK任务进度: 0%| | 0/8 [00:00<?, ?it/s][1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[WARNING][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected enum - serialized value may not be as expected [field_name='cover_task', input_value=0, input_type=int])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:52:10][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks over!!!, case_config_name:matmul_add_fp16, all_cases_num: 8[0m[0m

Serving Flask app 'sqlite_web.sqlite_web'
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on all addresses (0.0.0.0)
Running on http://127.0.0.1:8881
Running on http://192.168.9.148:8881
Press CTRL+C to quit
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823327][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823331][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823334][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823327][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823329][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823334][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823331][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823329][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823327][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823334][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823331][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823334][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823329][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823327][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823329][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823331][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m

ATK任务进度: 0%| | 0/8 [00:01<?, ?it/s][1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823352][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823350][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823354][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823357][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:12][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:02<?, ?it/s][1;36m[2026-09-22 12:52:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36m__init__.py[0m:[1;36m33[0m] [1m[34mLoaded mindspeed_ops library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/mindspeed_ops_C.cpython-312-aarch64-linux-gnu.so[0m[0m
[1;36m[2026-09-22 12:52:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36m__init__.py[0m:[1;36m92[0m] [1m[34mPreloaded custom op api library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/_cann_ops_custom/vendors/mindspeed_transformer/op_api/lib/libcust_opapi.so[0m[0m

ATK任务进度: 0%| | 0/8 [00:03<?, ?it/s][1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[823343][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:04<?, ?it/s][1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:52:14][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] use {'single_bm': {'type': 'high_precision', 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008, 'fp32_error': 0.00025, 'fp32_eb': 0.00025}} Standard.[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[823373][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[823375][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[823365][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[823377][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] SUCCESS clean cpu output data[0m[0m

ATK任务进度: 0%| | 0/8 [00:05<?, ?it/s][1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_0 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_1 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_2 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_3 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_4 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_5 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_6 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_7 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart write report ...[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+----------+------------------+------------------+--------------+----------------------+--------+--------------+
| 名称 | 总用例数 | 执行成功用例个数 | 执行失败用例个数 | 通过用例个数 | 错误信息匹配用例个数 | 通过率 | 精度是否达标 |
+-------+----------+------------------+------------------+--------------+----------------------+--------+--------------+
| cpu_0 | 8 | 8 | 0 | 8 | 0 | 100.0 | Pass |
+-------+----------+------------------+------------------+--------------+----------------------+--------+--------------+[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mTotal Task: 8, success 8, failed 0[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mSummary info: acc_pass_result:Pass is_acc_dc_pass:None is_e2e_perf_pass:None is_benchmark_perf_pass:None is_device_perf_pass:None is_device_memory_pass:None is_memory_check_pass:None .[0m[0m
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[823386][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34msave result excel file: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-52-09-180207/report/all_matmul_add_fp16_reports_2026-09-22-12-52-10.xlsx[0m[0m

ATK任务进度: 100%|██████████| 8/8 [00:05<00:00, 1.54it/s]
[1;36m[2026-09-22 12:52:15][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34matk task success![0m[0m

worker: Warm shutdown (MainProcess)

worker: Warm shutdown (MainProcess)
Restoring 4 unacknowledged message(s)
[1;36m[2026-09-22 12:52:19][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[821161][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34matk task finish. cost time: 9 s[0m[0m

Liz

27 天前评论：

accuracy_fp32_npu_vs_cpu.log

tensorflow is not install in env
tensorflow is not install in env
tensorflow is not install in env
ct-tool not installed, please install it first.
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mtask_output_path: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-51-44-056508[0m[0m
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mstart atk task: namespace(case='result/matmul_add_fp32/json/all_matmul_add_fp32.json', task=['accuracy'], plugin_path='atb_matmul_add.py', fast_case=None, slice_input=None, slice_input_ratio=None, nodes=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', white_list=None, black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, single_process=False, mssanitizer=False, ms_log_path=None, max_task=100, trace=False, timeout=None, db_timeout=None, db_path=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, atb_api_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, bm_output_path_adr='', sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, get_tiling_key_path=None, need_time=True, ct_tool=False, gm_init_flag=False, fluctuation_check=False, clean_l2_cache=False, nodes_list=[{'backend': <NodeType.NPU: 'npu'>, 'devices': [0], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}, {'backend': <NodeType.CPU: 'cpu'>, 'name': None, 'devices': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}])[0m[0m
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36matb_matmul_add.py[0m:[1;36m38[0m] [1m[34mmatmul_add_atk is already registered, repeated registration will coverage it, new module path is atb_matmul_add[0m[0m
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[816998][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[816998][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@create_dataset', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817000][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:44][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817002][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_0', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817004][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817006][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[817020][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[817020][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@cpu_run', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[817033][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[817033][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@compare', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks start!!!, case_config_name:matmul_add_fp32, all_cases_num: 8[0m[0m

ATK任务进度: 0%| | 0/8 [00:00<?, ?it/s][1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[WARNING][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected enum - serialized value may not be as expected [field_name='cover_task', input_value=0, input_type=int])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:51:45][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks over!!!, case_config_name:matmul_add_fp32, all_cases_num: 8[0m[0m

Serving Flask app 'sqlite_web.sqlite_web'
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on all addresses (0.0.0.0)
Running on http://127.0.0.1:8881
Running on http://192.168.9.148:8881
Press CTRL+C to quit
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817000][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817004][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817002][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817004][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817000][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817002][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817006][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817006][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817000][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817002][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817004][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817002][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817000][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817006][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817006][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817004][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m

ATK任务进度: 0%| | 0/8 [00:01<?, ?it/s][1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817027][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817025][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] Use Single bm standard, inputs will set to upper dtype (Only support fp16/bf16)[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817030][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:46][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run CPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:47][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:02<?, ?it/s][1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36m__init__.py[0m:[1;36m33[0m] [1m[34mLoaded mindspeed_ops library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/mindspeed_ops_C.cpython-312-aarch64-linux-gnu.so[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36m__init__.py[0m:[1;36m92[0m] [1m[34mPreloaded custom op api library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/_cann_ops_custom/vendors/mindspeed_transformer/op_api/lib/libcust_opapi.so[0m[0m

ATK任务进度: 0%| | 0/8 [00:03<?, ?it/s][1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATB_HOME_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1[0m[0m
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py310_cpu/npu_matmul_add_fp32/build.ninja...
Building extension module npu_matmul_add_fp32...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module npu_matmul_add_fp32...
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ASDOPS_HOME_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ASDOPS_OPS_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1/ops[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:48][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['accuracy'][0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[817013][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:04<?, ?it/s][1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start POST PROCESS, task:['accuracy'].[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] use {'single_bm': {'type': 'high_precision', 'fp32_error': 0.00025, 'fp32_eb': 0.00025, 'fp16_error': 0.002, 'fp16_eb': 0.002, 'bf16_error': 0.008, 'bf16_eb': 0.008}} Standard.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:49][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] output_0 dtype is torch.float32, use SINGLE BM high_precision(单标杆) original precision.[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[817041][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 7] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[817047][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 5] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[817045][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 6] SUCCESS clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] start clean cpu output data[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[817043][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 4] SUCCESS clean cpu output data[0m[0m

ATK任务进度: 0%| | 0/8 [00:05<?, ?it/s][1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_0 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_1 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_2 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_3 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_4 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_5 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_6 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| id_7 | npu_0_input_bytes | cpu_0_input_bytes | npu_0_output_bytes | cpu_0_output_bytes | accuracy |
+-------+-------------------+-------------------+--------------------+--------------------+----------+
| npu_0 | None | None | None | None | - |
| cpu_0 | | | | | True |
+-------+-------------------+-------------------+--------------------+--------------------+----------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart write report ...[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+----------+------------------+------------------+--------------+----------------------+--------+--------------+
| 名称 | 总用例数 | 执行成功用例个数 | 执行失败用例个数 | 通过用例个数 | 错误信息匹配用例个数 | 通过率 | 精度是否达标 |
+-------+----------+------------------+------------------+--------------+----------------------+--------+--------------+
| cpu_0 | 8 | 8 | 0 | 8 | 0 | 100.0 | Pass |
+-------+----------+------------------+------------------+--------------+----------------------+--------+--------------+[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mTotal Task: 8, success 8, failed 0[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mSummary info: acc_pass_result:Pass is_acc_dc_pass:None is_e2e_perf_pass:None is_benchmark_perf_pass:None is_device_perf_pass:None is_device_memory_pass:None is_memory_check_pass:None .[0m[0m
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[ResultProcess-5][0m [1;36m[817056][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34msave result excel file: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-51-44-056508/report/all_matmul_add_fp32_reports_2026-09-22-12-51-45.xlsx[0m[0m

ATK任务进度: 100%|██████████| 8/8 [00:05<00:00, 1.55it/s]
[1;36m[2026-09-22 12:51:50][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34matk task success![0m[0m

worker: Warm shutdown (MainProcess)

worker: Warm shutdown (MainProcess)
Restoring 4 unacknowledged message(s)
[1;36m[2026-09-22 12:51:54][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[815156][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34matk task finish. cost time: 9 s[0m[0m

Liz

27 天前评论：

memory_device_fp16_fused_vs_decomp.log

tensorflow is not install in env
tensorflow is not install in env
tensorflow is not install in env
ct-tool not installed, please install it first.
[1;36m[2026-09-22 12:57:59][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mtask_output_path: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-57-59-103200[0m[0m
[1;36m[2026-09-22 12:57:59][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mstart atk task: namespace(case='result/matmul_add_fp16/json/all_matmul_add_fp16.json', task=['memory_device'], plugin_path='atb_matmul_add.py', fast_case=None, slice_input=None, slice_input_ratio=None, nodes=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', white_list=None, black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, single_process=False, mssanitizer=False, ms_log_path=None, max_task=100, trace=False, timeout=None, db_timeout=None, db_path=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, atb_api_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, bm_output_path_adr='', sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, get_tiling_key_path=None, need_time=True, ct_tool=False, gm_init_flag=False, fluctuation_check=False, clean_l2_cache=False, nodes_list=[{'backend': <NodeType.NPU: 'npu'>, 'devices': [0], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}, {'backend': <NodeType.NPU: 'npu'>, 'devices': [1], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}])[0m[0m
[1;36m[2026-09-22 12:57:59][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36matb_matmul_add.py[0m:[1;36m38[0m] [1m[34mmatmul_add_atk is already registered, repeated registration will coverage it, new module path is atb_matmul_add[0m[0m
[1;36m[2026-09-22 12:57:59][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[900270][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:57:59][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[900270][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@create_dataset', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900272][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900275][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_0', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900346][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900363][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_1', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[900394][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[900394][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@cpu_run', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900396][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900398][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[900402][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[900402][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@compare', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900400][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900403][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900411][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900413][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900415][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks start!!!, case_config_name:matmul_add_fp16, all_cases_num: 8[0m[0m

ATK任务进度: 0%| | 0/8 [00:00<?, ?it/s][1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900417][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[WARNING][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected enum - serialized value may not be as expected [field_name='cover_task', input_value=0, input_type=int])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:00][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[898379][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks over!!!, case_config_name:matmul_add_fp16, all_cases_num: 8[0m[0m

Serving Flask app 'sqlite_web.sqlite_web'
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on all addresses (0.0.0.0)
Running on http://127.0.0.1:8881
Running on http://192.168.9.148:8881
Press CTRL+C to quit
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900272][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900363][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900275][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900272][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900346][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900363][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900275][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900346][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900272][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900363][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900275][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900363][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900346][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900272][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[900346][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:58:01][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900275][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS CREATE DATASET.[0m[0m

ATK任务进度: 0%| | 0/8 [00:01<?, ?it/s][1;36m[2026-09-22 12:58:02][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:02][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:02<?, ?it/s][1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36m__init__.py[0m:[1;36m33[0m] [1m[34mLoaded mindspeed_ops library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/mindspeed_ops_C.cpython-312-aarch64-linux-gnu.so[0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36m__init__.py[0m:[1;36m92[0m] [1m[34mPreloaded custom op api library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/_cann_ops_custom/vendors/mindspeed_transformer/op_api/lib/libcust_opapi.so[0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:03][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m

ATK任务进度: 0%| | 0/8 [00:03<?, ?it/s]
ATK任务进度: 0%| | 0/8 [00:04<?, ?it/s][1;36m[2026-09-22 12:58:04][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 7168, output_bytes: 1024.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.01123046875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 6.33984375 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.15869140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 7168, output_bytes: 1024.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.01171875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 6.09765625 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 7168, output_bytes: 1024.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.01123046875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.1591796875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.15869140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 7168, output_bytes: 1024.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 458752, output_bytes: 131072.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.68994140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...12], 'stride': [512, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...12], 'stride': [512, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.01171875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 458752, output_bytes: 131072.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.68994140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...12], 'stride': [512, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...12], 'stride': [512, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.1591796875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 1835008, output_bytes: 524288.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 2.75244140625 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 20.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...56], 'stride': [256, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...56], 'stride': [256, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 458752, output_bytes: 131072.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.6904296875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...12], 'stride': [512, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...12], 'stride': [512, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 1835008, output_bytes: 524288.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 2.75244140625 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...56], 'stride': [256, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...56], 'stride': [256, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[900377][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 458752, output_bytes: 131072.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 0.6904296875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...12], 'stride': [512, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...12], 'stride': [512, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 1835008, output_bytes: 524288.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 2.7529296875 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 20.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...56], 'stride': [256, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...56], 'stride': [256, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 1835008, output_bytes: 524288.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, max pta memory allocated of npu: 2.7529296875 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp16, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...56], 'stride': [256, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...56], 'stride': [256, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[900362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:05<?, ?it/s][1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900413][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[900413][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900411][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[900411][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:58:05][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[900417][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start POST PROCESS, task:['performance_device

Liz

27 天前评论：

memory_device_fp32_fused_vs_decomp.log

tensorflow is not install in env
tensorflow is not install in env
tensorflow is not install in env
ct-tool not installed, please install it first.
[1;36m[2026-09-22 12:54:49][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mtask_output_path: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-54-49-272740[0m[0m
[1;36m[2026-09-22 12:54:49][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mstart atk task: namespace(case='result/matmul_add_fp32/json/all_matmul_add_fp32.json', task=['memory_device'], plugin_path='atb_matmul_add.py', fast_case=None, slice_input=None, slice_input_ratio=None, nodes=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', white_list=None, black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, single_process=False, mssanitizer=False, ms_log_path=None, max_task=100, trace=False, timeout=None, db_timeout=None, db_path=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, atb_api_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, bm_output_path_adr='', sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, get_tiling_key_path=None, need_time=True, ct_tool=False, gm_init_flag=False, fluctuation_check=False, clean_l2_cache=False, nodes_list=[{'backend': <NodeType.NPU: 'npu'>, 'devices': [0], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}, {'backend': <NodeType.NPU: 'npu'>, 'devices': [1], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}])[0m[0m
[1;36m[2026-09-22 12:54:49][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36matb_matmul_add.py[0m:[1;36m38[0m] [1m[34mmatmul_add_atk is already registered, repeated registration will coverage it, new module path is atb_matmul_add[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[858360][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[858360][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@create_dataset', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858362][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858364][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_0', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858366][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858368][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_1', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[858469][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[858469][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@cpu_run', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858472][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858474][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[858479][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[858479][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@compare', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858476][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858478][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks start!!!, case_config_name:matmul_add_fp32, all_cases_num: 8[0m[0m

ATK任务进度: 0%| | 0/8 [00:00<?, ?it/s][1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[WARNING][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected enum - serialized value may not be as expected [field_name='cover_task', input_value=0, input_type=int])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:50][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[856599][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks over!!!, case_config_name:matmul_add_fp32, all_cases_num: 8[0m[0m

Serving Flask app 'sqlite_web.sqlite_web'
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on all addresses (0.0.0.0)
Running on http://127.0.0.1:8881
Running on http://192.168.9.148:8881
Press CTRL+C to quit
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858364][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858368][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858364][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858368][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858366][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858366][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858364][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858368][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858368][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858364][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858366][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858366][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:54:51][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858362][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS CREATE DATASET.[0m[0m

ATK任务进度: 0%| | 0/8 [00:01<?, ?it/s][1;36m[2026-09-22 12:54:52][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:52][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:02<?, ?it/s][1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36m__init__.py[0m:[1;36m33[0m] [1m[34mLoaded mindspeed_ops library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/mindspeed_ops_C.cpython-312-aarch64-linux-gnu.so[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36m__init__.py[0m:[1;36m92[0m] [1m[34mPreloaded custom op api library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/_cann_ops_custom/vendors/mindspeed_transformer/op_api/lib/libcust_opapi.so[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATB_HOME_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1[0m[0m
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
Emitting ninja build file /root/.cache/torch_extensions/py310_cpu/npu_matmul_add_fp32/build.ninja...
Building extension module npu_matmul_add_fp32...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module npu_matmul_add_fp32...
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m

ATK任务进度: 0%| | 0/8 [00:03<?, ?it/s][1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 8192, output_bytes: 2048.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ASDOPS_HOME_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ASDOPS_OPS_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1/ops[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.01171875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 104.140625 MB.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:53][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.189453125 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 8192, output_bytes: 2048.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.01171875 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.189453125 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 589824, output_bytes: 262144.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.814453125 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...12], 'stride': [512, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...12], 'stride': [512, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 589824, output_bytes: 262144.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.814453125 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 2359296, output_bytes: 1048576.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 3.251953125 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 20.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...56], 'stride': [256, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...56], 'stride': [256, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 2359296, output_bytes: 1048576.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 3.251953125 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:54][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[858375][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:04<?, ?it/s][1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 8192, output_bytes: 2048.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.01318359375 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 6.12109375 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.22119140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 8192, output_bytes: 2048.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.01318359375 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: -0.22265625 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.22119140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 589824, output_bytes: 262144.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.93994140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...12], 'stride': [512, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...12], 'stride': [512, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 589824, output_bytes: 262144.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 0.93994140625 MB, pta memory reserved of npu: 2.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 2359296, output_bytes: 1048576.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 3.75244140625 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 20.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...56], 'stride': [256, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...56], 'stride': [256, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] Start Execute OPP on ['performance_device_pta'][0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 2359296, output_bytes: 1048576.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, max pta memory allocated of npu: 3.75244140625 MB, pta memory reserved of npu: 22.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mkernel name: matmul_add_fp32, memory used by npu: 0.0 MB.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[858466][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:05<?, ?it/s][1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 1] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 3] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:55][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[858487][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[858497][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[858500][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[858491][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start POST PROCESS, task:['performance_device_pta'].[0m[0m
[1;36m[2026-09-22 12:54:56][0m [1;36m[INFO][0m [1;36m

Liz

27 天前评论：

performance_device_fp16_fused_vs_decomp.log

tensorflow is not install in env
tensorflow is not install in env
tensorflow is not install in env
ct-tool not installed, please install it first.
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mtask_output_path: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mstart atk task: namespace(case='result/matmul_add_fp16/json/all_matmul_add_fp16.json', task=['performance_device'], plugin_path='atb_matmul_add.py', fast_case=None, slice_input=None, slice_input_ratio=None, nodes=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', white_list=None, black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, single_process=False, mssanitizer=False, ms_log_path=None, max_task=100, trace=False, timeout=None, db_timeout=None, db_path=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, atb_api_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, bm_output_path_adr='', sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, get_tiling_key_path=None, need_time=True, ct_tool=False, gm_init_flag=False, fluctuation_check=False, clean_l2_cache=False, nodes_list=[{'backend': <NodeType.NPU: 'npu'>, 'devices': [0], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}, {'backend': <NodeType.NPU: 'npu'>, 'devices': [1], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}])[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36matb_matmul_add.py[0m:[1;36m38[0m] [1m[34mmatmul_add_atk is already registered, repeated registration will coverage it, new module path is atb_matmul_add[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[875796][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[875796][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@create_dataset', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[875798][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[875800][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_0', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[875802][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[875805][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_1', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[875948][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[875948][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@cpu_run', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[875963][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[875965][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[875969][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[875969][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@compare', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[875967][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[875971][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[876007][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[876020][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks start!!!, case_config_name:matmul_add_fp16, all_cases_num: 8[0m[0m

ATK任务进度: 0%| | 0/8 [00:00<?, ?it/s][1;36m[2026-09-22 12:56:18][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[WARNING][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected enum - serialized value may not be as expected [field_name='cover_task', input_value=0, input_type=int])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[873795][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks over!!!, case_config_name:matmul_add_fp16, all_cases_num: 8[0m[0m

Serving Flask app 'sqlite_web.sqlite_web'
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on all addresses (0.0.0.0)
Running on http://127.0.0.1:8881
Running on http://192.168.9.148:8881
Press CTRL+C to quit
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[875798][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[875802][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[875798][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[875805][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[875802][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[875805][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[875800][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[875800][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[875798][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[875802][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[875805][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[875802][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[875798][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[875800][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[875800][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:56:19][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[875805][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS CREATE DATASET.[0m[0m

ATK任务进度: 0%| | 0/8 [00:01<?, ?it/s][1;36m[2026-09-22 12:56:20][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:20][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:02<?, ?it/s][1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36m__init__.py[0m:[1;36m33[0m] [1m[34mLoaded mindspeed_ops library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/mindspeed_ops_C.cpython-312-aarch64-linux-gnu.so[0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36m__init__.py[0m:[1;36m92[0m] [1m[34mPreloaded custom op api library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/_cann_ops_custom/vendors/mindspeed_transformer/op_api/lib/libcust_opapi.so[0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m

ATK任务进度: 0%| | 0/8 [00:03<?, ?it/s]
ATK任务进度: 0%| | 0/8 [00:04<?, ?it/s][1;36m[2026-09-22 12:56:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 7168, output_bytes: 1024.[0m[0m
[1;36m[2026-09-22 12:56:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m
[1;36m[2026-09-22 12:56:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 7168, output_bytes: 1024.[0m[0m
[1;36m[2026-09-22 12:56:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m

ATK任务进度: 0%| | 0/8 [00:05<?, ?it/s]
ATK任务进度: 0%| | 0/8 [00:06<?, ?it/s][1;36m[2026-09-22 12:56:25][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:56:25] [WARNING] [875804] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[1;36m[2026-09-22 12:56:25][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:56:25] [WARNING] [875889] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.

ATK任务进度: 0%| | 0/8 [00:07<?, ?it/s][2026-09-22 12:56:26] [INFO] [877740] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549/profile/npu_0/all_matmul_add_fp16/0/admin123_875804_20260922125625149_ascend_pt
[2026-09-22 12:56:26] [INFO] [877752] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549/profile/npu_1/all_matmul_add_fp16/0/admin123_875889_20260922125625203_ascend_pt
||
ATK任务进度: 0%| | 0/8 [00:08<?, ?it/s]//
ATK任务进度: 0%| | 0/8 [00:09<?, ?it/s]--
ATK任务进度: 0%| | 0/8 [00:10<?, ?it/s]\\[2026-09-22 12:56:29] [INFO] [877846] profiler.py: CANN profiling data parsed in a total time of 0:00:03.030960
[2026-09-22 12:56:29] [INFO] [877843] profiler.py: CANN profiling data parsed in a total time of 0:00:03.136916

ATK任务进度: 0%| | 0/8 [00:11<?, ?it/s]||[2026-09-22 12:56:30] [INFO] [877752] profiler.py: All profiling data parsed in a total time of 0:00:04.680525
[2026-09-22 12:56:30] [INFO] [877740] profiler.py: All profiling data parsed in a total time of 0:00:04.827651

ATK任务进度: 0%| | 0/8 [00:12<?, ?it/s][1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 7.094594594594593, iqr: 0.10499999999999998, media: 1.48[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 4.9180327868852505, iqr: 0.06000000000000005, media: 1.22[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] End to parses training profiler. Duration:0.01 second.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 2.7009999999999996, aicore_time: 0.9433999999999999, aic_mac_ratio: 0.04205000000000001[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 4.481213374698377, iqr: 0.06499999999999995, media: 1.4505[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 3.7815126050420296, iqr: 0.04500000000000015, media: 1.19[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] End to parses training profiler. Duration:0.01 second.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 2.6521, aicore_time: 0.9189999999999999, aic_mac_ratio: 0.04310000000000001[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:56:31][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m

ATK任务进度: 0%| | 0/8 [00:13<?, ?it/s][1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 2.701, 'read_bytes(MB)': 0.0068359375, 'write_bytes(MB)': 0.0009765625, 'io_bytes(MB)': 0.0078125, 'mem_bw(GB/s)': 2.824655509534, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 0.176540969346, 'mem_peak(MB)': 0.009765625, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.1, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] NPU AVG Profiler Time:2.7009999999999996 us, max memory allocated: 0 MB, max memory reserved: 128.38671875 MB, NPU std_deviation: 0.06803518032529669, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 2.65205, 'read_bytes(MB)': 0.0068359375, 'write_bytes(MB)': 0.0009765625, 'io_bytes(MB)': 0.0078125, 'mem_bw(GB/s)': 2.876791361871, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 0.179799460117, 'mem_peak(MB)': 0.009765625, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.2, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] NPU AVG Profiler Time:2.65205 us, max memory allocated: 0 MB, max memory reserved: 128.38671875 MB, NPU std_deviation: 0.06363738540493774, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m
[1;36m[2026-09-22 12:56:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m

ATK任务进度: 0%| | 0/8 [00:14<?, ?it/s]
ATK任务进度: 0%| | 0/8 [00:15<?, ?it/s][1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start POST PROCESS, task:['performance_device'].[0m[0m
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[876032][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:56:34] [WARNING] [875889] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[1;36m[2026-09-22 12:56:34][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:56:34] [WARNING] [875804] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.

ATK任务进度: 0%| | 0/8 [00:16<?, ?it/s][1;36m[2026-09-22 12:56:35][0m [1;36m[INFO][0m [1;36m[ResultProcess-6][0m [1;36m[876039][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| id_0 | npu_0_input_bytes | npu_1_input_bytes | npu_0_output_bytes | npu_1_output_bytes | npu_0_std_deviation | npu_1_std_deviation | npu_0_cube_computation | npu_1_cube_computation | npu_0_api_statistic | npu_1_api_statistic | npu_0_fluctuation_result | npu_1_fluctuation_result | device_perf(us) | perf_device_result |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| npu_0 | 7168.0 | 7168.0 | 1024.0 | 1024.0 | 0.06363738540493774 | 0.06803518032529669 | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | None | None | None | None | 2.652050 | - |
| npu_1 | | | | | | | | | | | | | 2.701000 | True |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+[0m[0m
[2026-09-22 12:56:35] [INFO] [880183] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549/profile/npu_1/all_matmul_add_fp16/2/admin123_875889_20260922125634562_ascend_pt
[2026-09-22 12:56:35] [INFO] [880190] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549/profile/npu_0/all_matmul_add_fp16/2/admin123_875804_20260922125634598_ascend_pt
||
ATK任务进度: 12%|█▎ | 1/8 [00:17<00:07, 1.04s/it]//--\\[2026-09-22 12:56:38] [INFO] [880208] profiler.py: CANN profiling data parsed in a total time of 0:00:03.039003
[2026-09-22 12:56:38] [INFO] [880214] profiler.py: CANN profiling data parsed in a total time of 0:00:03.032450
||[2026-09-22 12:56:40] [INFO] [880183] profiler.py: All profiling data parsed in a total time of 0:00:04.657763
[2026-09-22 12:56:40] [INFO] [880190] profiler.py: All profiling data parsed in a total time of 0:00:04.730787
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 18.48739495798319, iqr: 0.43999999999999995, media: 2.38[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 4.663212435233153, iqr: 0.08999999999999986, media: 1.93[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 4.4239999999999995, aicore_time: 1.7967499999999998, aic_mac_ratio: 0.09920000000000002[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 17.517624439222395, iqr: 0.41000000000000014, media: 2.3405[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 7.591623036649205, iqr: 0.1449999999999998, media: 1.91[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 4.376150000000001, aicore_time: 1.7791500000000002, aic_mac_ratio: 0.10020000000000003[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:56:40][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 4.424, 'read_bytes(MB)': 0.09375, 'write_bytes(MB)': 0.03125, 'io_bytes(MB)': 0.125, 'mem_bw(GB/s)': 27.592746948463, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 1.724546684279, 'mem_peak(MB)': 0.126953125, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.3, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] NPU AVG Profiler Time:4.4239999999999995 us, max memory allocated: 0 MB, max memory reserved: 132.51171875 MB, NPU std_deviation: 0.15681460683470988, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 4.3761, 'read_bytes(MB)': 0.09375, 'write_bytes(MB)': 0.03125, 'io_bytes(MB)': 0.125, 'mem_bw(GB/s)': 27.894772171568, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 1.743423260723, 'mem_peak(MB)': 0.126953125, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.3, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] NPU AVG Profiler Time:4.376099999999999 us, max memory allocated: 0 MB, max memory reserved: 132.51171875 MB, NPU std_deviation: 0.1643585532225245, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.bfloat16...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 98304, output_bytes: 32768.[0m[0m
[1;36m[2026-09-22 12:56:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start POST PROCESS, task:['performance_device'].[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[876024][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:56:43][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:56:43] [WARNING] [875889] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[1;36m[2026-09-22 12:56:44][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:56:44] [WARNING] [875804] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[1;36m[2026-09-22 12:56:44][0m [1;36m[INFO][0m [1;36m[ResultProcess-6][0m [1;36m[876039][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| id_2 | npu_0_input_bytes | npu_1_input_bytes | npu_0_output_bytes | npu_1_output_bytes | npu_0_std_deviation | npu_1_std_deviation | npu_0_cube_computation | npu_1_cube_computation | npu_0_api_statistic | npu_1_api_statistic | npu_0_fluctuation_result | npu_1_fluctuation_result | device_perf(us) | perf_device_result |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| npu_0 | 98304.0 | 98304.0 | 32768.0 | 32768.0 | 0.1643585532225245 | 0.15681460683470988 | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | None | None | None | None | 4.376100 | - |
| npu_1 | | | | | | | | | | | | | 4.424000 | True |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+[0m[0m

ATK任务进度: 25%|██▌ | 2/8 [00:25<00:30, 5.15s/it][2026-09-22 12:56:44] [INFO] [882840] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549/profile/npu_1/all_matmul_add_fp16/3/admin123_875889_20260922125643824_ascend_pt
|[2026-09-22 12:56:44] [INFO] [882933] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp16_2026-09-22-12-56-17-404549/profile/npu_0/all_matmul_add_fp16/3/admin123_875804_20260922125643931_ascend_pt
|//--\\[2026-09-22 12:56:47] [INFO] [882953] profiler.py: CANN profiling data parsed in a total time of 0:00:03.025461
[2026-09-22 12:56:48] [INFO] [882959] profiler.py: CANN profiling data parsed in a total time of 0:00:03.047567
||[2026-09-22 12:56:49] [INFO] [882840] profiler.py: All profiling data parsed in a total time of 0:00:04.674657
[2026-09-22 12:56:49] [INFO] [882933] profiler.py: All profiling data parsed in a total time of 0:00:04.685068
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 14.90566037735849, iqr: 0.395, media: 2.6500000000000004[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 3.821656050955417, iqr: 0.06000000000000005, media: 1.57[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 4.093, aicore_time: 1.7997, aic_mac_ratio: 0.09885000000000001[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:56:49][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 18.66952789699571, iqr: 0.43500000000000005, media: 2.33[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 5.400641025641021, iqr: 0.08424999999999994, media: 1.56[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 4.03815, aicore_time: 1.77575, aic_mac_ratio: 0.10020000000000004[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:56:50][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[875804][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 4.093, 'read_bytes(MB)': 0.09375, 'write_bytes(MB)': 0.03125, 'io_bytes(MB)': 0.125, 'mem_bw(GB/s)': 29.824166259467, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 1.864010391217, 'mem_peak(MB)': 0.126953125, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.5, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] NPU AVG Profiler Time:4.093 us, max memory allocated: 0 MB, max memory reserved: 140.38671875 MB, NPU std_deviation: 0.126727421631374, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float16'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[875889][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:56:51][0m [1;36m[INFO][

Liz

27 天前评论：

performance_device_fp32_fused_vs_decomp.log

tensorflow is not install in env
tensorflow is not install in env
tensorflow is not install in env
ct-tool not installed, please install it first.
[1;36m[2026-09-22 12:53:09][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mtask_output_path: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527[0m[0m
[1;36m[2026-09-22 12:53:09][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mstart atk task: namespace(case='result/matmul_add_fp32/json/all_matmul_add_fp32.json', task=['performance_device'], plugin_path='atb_matmul_add.py', fast_case=None, slice_input=None, slice_input_ratio=None, nodes=None, log='info', celery_log='warning', save_data=None, print_data='', start=None, end=None, acl_json='', white_list=None, black_list=None, random_nums=None, random_align_32b_nums=None, bm_device=<BenchmarkDevice.CPU: 'cpu'>, syc_dataset=False, single_process=False, mssanitizer=False, ms_log_path=None, max_task=100, trace=False, timeout=None, db_timeout=None, db_path=None, xrun_whl=None, bind_cpu_type='1', boundary_filter='default', keyword_filter=None, cpp_func_signature_type_path=False, triton_ut_path=None, atb_api_path=None, input_data=None, experimental_config_path=None, bm_output_path=None, bm_output_path_adr='', sqlite_web_port='8881', loop_nums=1, disable_id_seed=False, dc_loop_nums=50, performance_data=[], atc_config=None, get_tiling_key_path=None, need_time=True, ct_tool=False, gm_init_flag=False, fluctuation_check=False, clean_l2_cache=False, nodes_list=[{'backend': <NodeType.NPU: 'npu'>, 'devices': [0], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}, {'backend': <NodeType.NPU: 'npu'>, 'devices': [1], 'name': None, 'host': '127.0.0.1', 'port': 9090, 'task': None, 'output_path': None, 'run_modes': None, 'is_compare': True, 'is_bm': False, 'is_dist': False, 'dist_backend': 'hccl', 'bm_file': None, 'fusion_flag': False}])[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36matb_matmul_add.py[0m:[1;36m38[0m] [1m[34mmatmul_add_atk is already registered, repeated registration will coverage it, new module path is atb_matmul_add[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[834471][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@create_dataset][0m [1;36m[834471][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@create_dataset', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834473][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834475][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_0', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834477][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834479][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@device_run_1', 'pool': 'solo', 'concurrency': 1, 'autoscale': '1,1', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[834499][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@cpu_run][0m [1;36m[834499][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@cpu_run', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834501][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834503][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[834514][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@compare][0m [1;36m[834514][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34mstart run {'queues': '127.0.0.1:9090@compare', 'pool': 'prefork', 'concurrency': 4, 'autoscale': '4,4', 'prefetch_multiplier': 16}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834508][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834513][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834522][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATK_CPU_INFO = {"cpu_ids_without_devices": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239], "device_cpu_ids": {"1": [243, 242], "0": [241, 240]}}[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834524][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834526][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks start!!!, case_config_name:matmul_add_fp32, all_cases_num: 8[0m[0m

ATK任务进度: 0%| | 0/8 [00:00<?, ?it/s][1;36m[2026-09-22 12:53:10][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env CELERY_LOG_LEVEL = 30[0m[0m
[1;36m[2026-09-22 12:53:10][0m [1;36m[WARNING][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected enum - serialized value may not be as expected [field_name='cover_task', input_value=0, input_type=int])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[MainProcess][0m [1;36m[832382][0m [[1;36mdecorators.py[0m:[1;36m34[0m] [1m[34mcreate_tasks over!!!, case_config_name:matmul_add_fp32, all_cases_num: 8[0m[0m

Serving Flask app 'sqlite_web.sqlite_web'
Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Running on all addresses (0.0.0.0)
Running on http://127.0.0.1:8881
Running on http://192.168.9.148:8881
Press CTRL+C to quit
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834473][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834479][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834473][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834477][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834479][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834475][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834477][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834475][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834473][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-1][0m [1;36m[834473][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 4] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834477][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834479][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-3][0m [1;36m[834477][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 5] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834475][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] start CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-2][0m [1;36m[834475][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 7] SUCCESS CREATE DATASET.[0m[0m
[1;36m[2026-09-22 12:53:11][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834479][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 6] SUCCESS CREATE DATASET.[0m[0m

ATK任务进度: 0%| | 0/8 [00:01<?, ?it/s][1;36m[2026-09-22 12:53:12][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:12][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start run NPU OPP TASK.[0m[0m

ATK任务进度: 0%| | 0/8 [00:02<?, ?it/s][1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36m__init__.py[0m:[1;36m33[0m] [1m[34mLoaded mindspeed_ops library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/mindspeed_ops_C.cpython-312-aarch64-linux-gnu.so[0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36m__init__.py[0m:[1;36m92[0m] [1m[34mPreloaded custom op api library: /home/l00661657/version-mul/018/MindSpeed-Ops/mindspeed_ops/_cann_ops_custom/vendors/mindspeed_transformer/op_api/lib/libcust_opapi.so[0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ATB_HOME_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1[0m[0m
Using /root/.cache/torch_extensions/py310_cpu as PyTorch extensions root...
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:13][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
Emitting ninja build file /root/.cache/torch_extensions/py310_cpu/npu_matmul_add_fp32/build.ninja...
Building extension module npu_matmul_add_fp32...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module npu_matmul_add_fp32...

ATK任务进度: 0%| | 0/8 [00:03<?, ?it/s][1;36m[2026-09-22 12:53:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 8192, output_bytes: 2048.[0m[0m
[1;36m[2026-09-22 12:53:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ASDOPS_HOME_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1[0m[0m
[1;36m[2026-09-22 12:53:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mpatch_getenv.py[0m:[1;36m15[0m] [1m[34mget env ASDOPS_OPS_PATH = /home/cann/CANN9.0.0/nnal/atb/latest/atb/cxx_abi_1/ops[0m[0m
[1;36m[2026-09-22 12:53:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m

ATK任务进度: 0%| | 0/8 [00:04<?, ?it/s][1;36m[2026-09-22 12:53:14][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 8192, output_bytes: 2048.[0m[0m
[1;36m[2026-09-22 12:53:15][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m

ATK任务进度: 0%| | 0/8 [00:05<?, ?it/s][1;36m[2026-09-22 12:53:16][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:53:16] [WARNING] [834486] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[2026-09-22 12:53:16] [INFO] [836064] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527/profile/npu_0/all_matmul_add_fp32/0/admin123_834486_20260922125315932_ascend_pt
|
ATK任务进度: 0%| | 0/8 [00:06<?, ?it/s][1;36m[2026-09-22 12:53:17][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:53:17] [WARNING] [834496] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
/[2026-09-22 12:53:17] [INFO] [836398] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527/profile/npu_1/all_matmul_add_fp32/0/admin123_834496_20260922125316898_ascend_pt

ATK任务进度: 0%| | 0/8 [00:07<?, ?it/s]|-
ATK任务进度: 0%| | 0/8 [00:08<?, ?it/s]/\
ATK任务进度: 0%| | 0/8 [00:09<?, ?it/s]-[2026-09-22 12:53:19] [INFO] [836122] profiler.py: CANN profiling data parsed in a total time of 0:00:03.032933
|
ATK任务进度: 0%| | 0/8 [00:10<?, ?it/s]\[2026-09-22 12:53:21] [INFO] [836450] profiler.py: CANN profiling data parsed in a total time of 0:00:03.029415
[2026-09-22 12:53:21] [INFO] [836064] profiler.py: All profiling data parsed in a total time of 0:00:04.659539

ATK任务进度: 0%| | 0/8 [00:11<?, ?it/s][1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 2.93918918918918, iqr: 0.06524999999999981, media: 2.22[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] End to parses training profiler. Duration:0.01 second.[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 2.2172000000000005, aicore_time: 1.6843, aic_mac_ratio: 0.023400000000000008[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:53:21][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
|[2026-09-22 12:53:22] [INFO] [836398] profiler.py: All profiling data parsed in a total time of 0:00:04.676586

ATK任务进度: 0%| | 0/8 [00:12<?, ?it/s][1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 10.000000000000002, iqr: 0.14500000000000002, media: 1.45[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 6.967213114754115, iqr: 0.08500000000000019, media: 1.22[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] End to parses training profiler. Duration:0.01 second.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 2.7009999999999996, aicore_time: 0.9362499999999999, aic_mac_ratio: 0.04235000000000001[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 2.2171, 'read_bytes(MB)': 0.0078125, 'write_bytes(MB)': 0.001953125, 'io_bytes(MB)': 0.009765625, 'mem_bw(GB/s)': 4.301449264383, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 0.268840579024, 'mem_peak(MB)': 0.01171875, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.3, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] NPU AVG Profiler Time:2.2171 us, max memory allocated: 0 MB, max memory reserved: 220.38671875 MB, NPU std_deviation: 0.07547423401399973, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:53:23][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m

ATK任务进度: 0%| | 0/8 [00:13<?, ?it/s][1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 2.701, 'read_bytes(MB)': 0.0078125, 'write_bytes(MB)': 0.001953125, 'io_bytes(MB)': 0.009765625, 'mem_bw(GB/s)': 3.530819386917, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 0.220676211682, 'mem_peak(MB)': 0.01171875, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 106.0, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] NPU AVG Profiler Time:2.7009999999999996 us, max memory allocated: 0 MB, max memory reserved: 128.38671875 MB, NPU std_deviation: 0.06989573941631713, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'... 32], 'stride': [32, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:53:24][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m

ATK任务进度: 0%| | 0/8 [00:14<?, ?it/s][1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] start POST PROCESS, task:['performance_device'].[0m[0m
[1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 0] SUCCESS POST PROCESS.[0m[0m
[1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 0] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:53:25] [WARNING] [834486] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.

ATK任务进度: 0%| | 0/8 [00:15<?, ?it/s][1;36m[2026-09-22 12:53:25][0m [1;36m[INFO][0m [1;36m[ResultProcess-6][0m [1;36m[834537][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| id_0 | npu_0_input_bytes | npu_1_input_bytes | npu_0_output_bytes | npu_1_output_bytes | npu_0_std_deviation | npu_1_std_deviation | npu_0_cube_computation | npu_1_cube_computation | npu_0_api_statistic | npu_1_api_statistic | npu_0_fluctuation_result | npu_1_fluctuation_result | device_perf(us) | perf_device_result |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| npu_0 | 8192.0 | 8192.0 | 2048.0 | 2048.0 | 0.07547423401399973 | 0.06989573941631713 | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | None | None | None | None | 2.217100 | - |
| npu_1 | | | | | | | | | | | | | 2.701000 | True |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+[0m[0m
[2026-09-22 12:53:26] [INFO] [838413] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527/profile/npu_0/all_matmul_add_fp32/2/admin123_834486_20260922125325231_ascend_pt
|[1;36m[2026-09-22 12:53:26][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:53:26] [WARNING] [834496] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.

ATK任务进度: 12%|█▎ | 1/8 [00:16<00:07, 1.04s/it]/[2026-09-22 12:53:27] [INFO] [838773] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527/profile/npu_1/all_matmul_add_fp32/2/admin123_834496_20260922125326287_ascend_pt
|-/\-[2026-09-22 12:53:29] [INFO] [838428] profiler.py: CANN profiling data parsed in a total time of 0:00:03.026928
|\[2026-09-22 12:53:30] [INFO] [838824] profiler.py: CANN profiling data parsed in a total time of 0:00:03.032829
[2026-09-22 12:53:30] [INFO] [838413] profiler.py: All profiling data parsed in a total time of 0:00:04.662676
|[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 3.237410071942425, iqr: 0.08999999999999941, media: 2.78[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 2.789, aicore_time: 2.2594499999999997, aic_mac_ratio: 0.14145000000000002[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:53:31][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[2026-09-22 12:53:31] [INFO] [838773] profiler.py: All profiling data parsed in a total time of 0:00:04.603438
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 14.102564102564125, iqr: 0.3300000000000005, media: 2.34[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 5.4973821989528675, iqr: 0.10499999999999976, media: 1.91[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 4.33, aicore_time: 1.7616000000000003, aic_mac_ratio: 0.10070000000000001[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 2.7891, 'read_bytes(MB)': 0.125, 'write_bytes(MB)': 0.0625, 'io_bytes(MB)': 0.1875, 'mem_bw(GB/s)': 65.650377810046, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 4.103148613128, 'mem_peak(MB)': 0.189453125, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.1, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] NPU AVG Profiler Time:2.7891 us, max memory allocated: 0 MB, max memory reserved: 220.51171875 MB, NPU std_deviation: 0.09663850164401344, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:53:32][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 4.33, 'read_bytes(MB)': 0.125, 'write_bytes(MB)': 0.0625, 'io_bytes(MB)': 0.1875, 'mem_bw(GB/s)': 42.287637124711, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 2.642977320294, 'mem_peak(MB)': 0.189453125, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 106.2, 'temp(C)': 48.0}[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] NPU AVG Profiler Time:4.33 us, max memory allocated: 0 MB, max memory reserved: 132.51171875 MB, NPU std_deviation: 0.12612914519568547, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] START Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Init by input data.[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] Start Execute OPP on ['performance_device'][0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34minput_bytes: 131072, output_bytes: 65536.[0m[0m
[1;36m[2026-09-22 12:53:33][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成 warmup 阶段==[0m[0m
[1;36m[2026-09-22 12:53:34][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:53:34] [WARNING] [834486] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[2026-09-22 12:53:35] [INFO] [841080] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527/profile/npu_0/all_matmul_add_fp32/3/admin123_834486_20260922125334625_ascend_pt
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] start POST PROCESS, task:['performance_device'].[0m[0m
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 2] SUCCESS POST PROCESS.[0m[0m
|[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] start clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ForkPoolWorker-4][0m [1;36m[834528][0m [[1;36mtask.py[0m:[1;36m411[0m] [1m[34m[case 2] SUCCESS clean npu output data[0m[0m
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m=完成自定义npu api阶段==[0m[0m
[2026-09-22 12:53:35] [WARNING] [834496] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[1;36m[2026-09-22 12:53:35][0m [1;36m[INFO][0m [1;36m[ResultProcess-6][0m [1;36m[834537][0m [[1;36mprocess.py[0m:[1;36m314[0m] [1m[34m
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| id_2 | npu_0_input_bytes | npu_1_input_bytes | npu_0_output_bytes | npu_1_output_bytes | npu_0_std_deviation | npu_1_std_deviation | npu_0_cube_computation | npu_1_cube_computation | npu_0_api_statistic | npu_1_api_statistic | npu_0_fluctuation_result | npu_1_fluctuation_result | device_perf(us) | perf_device_result |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+
| npu_0 | 131072.0 | 131072.0 | 65536.0 | 65536.0 | 0.09663850164401344 | 0.12612914519568547 | [0.0, 0.0, 0.0] | [0.0, 0.0, 0.0] | None | None | None | None | 2.789100 | - |
| npu_1 | | | | | | | | | | | | | 4.330000 | True |
+-------+-------------------+-------------------+--------------------+--------------------+---------------------+---------------------+------------------------+------------------------+---------------------+---------------------+--------------------------+--------------------------+-----------------+--------------------+[0m[0m

ATK任务进度: 25%|██▌ | 2/8 [00:25<00:34, 5.74s/it]/[2026-09-22 12:53:36] [INFO] [841467] profiler.py: Start parsing profiling data in sync mode at: /home/l00661657/version-mul/018/MindSpeed-Ops/tests/atk_tests/atb/matmul_add/atk_output/all_matmul_add_fp32_2026-09-22-12-53-09-260527/profile/npu_1/all_matmul_add_fp32/3/admin123_834496_20260922125335656_ascend_pt
|-/-\[2026-09-22 12:53:38] [INFO] [841092] profiler.py: CANN profiling data parsed in a total time of 0:00:03.043743
|\[2026-09-22 12:53:39] [INFO] [841531] profiler.py: CANN profiling data parsed in a total time of 0:00:03.030627
[2026-09-22 12:53:40] [INFO] [841080] profiler.py: All profiling data parsed in a total time of 0:00:04.766539
|[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 2.8776978417266053, iqr: 0.07999999999999963, media: 2.78[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] End to parses training profiler. Duration:0.00 second.[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 2.75805, aicore_time: 2.22695, aic_mac_ratio: 0.14354999999999998[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:53:40][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[2026-09-22 12:53:41] [INFO] [841467] profiler.py: All profiling data parsed in a total time of 0:00:04.691096
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] Start parsing Profiling files.[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 14.618644067796618, iqr: 0.3450000000000002, media: 2.36[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_check cqv: 4.450261780104711, iqr: 0.08499999999999996, media: 1.91[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mperf_fluctuation_result: True[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] End to parses training profiler. Duration:0.01 second.[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mtask_duration: 4.343999999999999, aicore_time: 1.77195, aic_mac_ratio: 0.10030000000000003[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mcal_cube_computation() is not custom defined, TFlops_e2e,TFlops_cube,MFU return 0,0,0 as default.[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport_custom_data() is not custom defined, return None.[0m[0m
[1;36m[2026-09-22 12:53:41][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_1][0m [1;36m[834496][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[33mexport_custom_data should return a dict, but get None, maybe not custom defined[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mchip model: Ascend910B3, cann_version: 9.0.0, hardware_computation: 295, hardware_hbm: 1600[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34mexport total custom data: {'latency(us)': 2.758, 'read_bytes(MB)': 0.125, 'write_bytes(MB)': 0.0625, 'io_bytes(MB)': 0.1875, 'mem_bw(GB/s)': 66.390670322698, 'calc_flops_power(FLOPs)': 0, 'calc_utilization(%)': 0.0, 'mem_utilization(%)': 4.149416895169, 'mem_peak(MB)': 0.189453125, 'atta_perf(TFLOPs/s)': 0.0, 'calc_mem_ratio(FLOPs/Byte)': 0.0, 'chip_model': 'Ascend910B3', 'cann_version': 'CANN9.0.0', 'pwr(W)': 97.2, 'temp(C)': 49.0}[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] NPU AVG Profiler Time:2.758 us, max memory allocated: 0 MB, max memory reserved: 228.38671875 MB, NPU std_deviation: 0.06511718283218329, TFlops_e2e: 0, TFlops_cube: 0, MFU: 0[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3][RUN OPP TASK] SUCCESS Execute OPP.[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[WARNING][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mwarnings.py[0m:[1;36m109[0m] [1m[33m/home/miniconda3/envs/lz_base/lib/python3.10/site-packages/pydantic/main.py:475: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected OutputData - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
PydanticSerializationUnexpectedValue(Expected list[definition-ref] - serialized value may not be as expected [field_name='output_info_list', input_value={'dtype': 'torch.float32'...28], 'stride': [128, 1]}, input_type=dict])
return self.pydantic_serializer.to_python(
[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 3] SUCCESS run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1] start run NPU OPP TASK.[0m[0m
[1;36m[2026-09-22 12:53:42][0m [1;36m[INFO][0m [1;36m[127.0.0.1:9090@device_run_0][0m [1;36m[834486][0m [[1;36mtrace.py[0m:[1;36m736[0m] [1m[34m[case 1][R

ascend-robot成员

26 天前关闭了 issue

ascend-robot成员

26 天前 issue状态由 TODO 改变为 DONE

ascend-robot成员

26 天前添加了label：resolved