| @@ -84,6 +84,7 @@ CANN算子开发者需要同时维护Host端和Kernel端代码。 | |||
| 84 | | 400-422 | Parallel Welford | 基数400 = Welford算法族的起始编号 | | 84 | | 400-422 | Parallel Welford | 基数400 = Welford算法族的起始编号 | |
| 85 | | 300-322 | Two-Pass + Binary累加 | 基数300 = TwoPass算法族的起始编号 | | 85 | | 300-322 | Two-Pass + Binary累加 | 基数300 = TwoPass算法族的起始编号 | |
| 86 | | 500-522 | Two-Pass Perf(优化版) | 基数500 = TwoPassPerf算法族的起始编号 | | 86 | | 500-522 | Two-Pass Perf(优化版) | 基数500 = TwoPassPerf算法族的起始编号 | |
| 87 | +| 800-822 | SplitR 跨核分段归约 | 基数800 = SplitR算法族的起始编号 | | ||
| 87 | 88 | ||
| 88 | 每个算法族内的取值范围(如400-422)覆盖了不同的**数据类型组合**(如`FLOAT_FLOAT`、`HALF_FLOAT`、`BF16_FLOAT`等),即每个具体的`(src_dtype, accum_dtype)`组合占用一个TILING_KEY值。因此`400-422`意味着Welford族覆盖了约23种数据类型组合。这些范围值的设定是CANN框架的命名约定,与底层实现无直接数学关联。 | 89 | 每个算法族内的取值范围(如400-422)覆盖了不同的**数据类型组合**(如`FLOAT_FLOAT`、`HALF_FLOAT`、`BF16_FLOAT`等),即每个具体的`(src_dtype, accum_dtype)`组合占用一个TILING_KEY值。因此`400-422`意味着Welford族覆盖了约23种数据类型组合。这些范围值的设定是CANN框架的命名约定,与底层实现无直接数学关联。 |
| 89 | 90 | ||
| @@ -98,7 +99,7 @@ if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) { ... } | |||
| 98 | 99 | ||
| 99 | **TILING_KEY与Tiling模板优先级的关系:** | 100 | **TILING_KEY与Tiling模板优先级的关系:** |
| 100 | 101 | ||
| 101 | -Tiling模板注册时附带一个优先级数值(如Welford为4000,TwoPass为200,TwoPassPerf为150)。框架按优先级从高到低尝试匹配:先检查Welford(4000)是否满足`IsCapable()`,不满足则尝试TwoPass(200),再不满足则尝试TwoPassPerf(150)。这与TILING_KEY的范围值(400/300/500)是两套独立的编号体系 —— 前者控制模板匹配顺序,后者是算法标识符。 | 102 | +Tiling模板注册时附带一个优先级数值(如SplitR为140,TwoPassPerf为150,TwoPass为200,Welford为4000)。框架按优先级数值从小到大依次尝试匹配:先检查SplitR(140)是否满足`IsCapable()`,不满足则尝试TwoPassPerf(150),再不满足则尝试TwoPass(200),最后尝试Welford(4000)。这与TILING_KEY的范围值(400/300/500/800)是两套独立的编号体系 —— 前者控制模板匹配顺序,后者是算法标识符。 |
| 102 | 103 | ||
| 103 | #### 0.3.4 regbase(寄存器基址寻址模式) | 104 | #### 0.3.4 regbase(寄存器基址寻址模式) |
| 104 | 105 | ||
| @@ -108,7 +109,7 @@ Tiling模板注册时附带一个优先级数值(如Welford为4000,TwoPass | |||
| 108 | - **支持向量融合**:部分CANN向量融合指令(如`FusedMulDstAdd`)要求在regbase模式下才能使用 | 109 | - **支持向量融合**:部分CANN向量融合指令(如`FusedMulDstAdd`)要求在regbase模式下才能使用 |
| 109 | - **减少UB访问冲突**:通过寄存器中转减少了UB的读取次数 | 110 | - **减少UB访问冲突**:通过寄存器中转减少了UB的读取次数 |
| 110 | 111 | ||
| 111 | -`isRegBase`是一个由Host端根据输入shape和数据类型计算出的布尔值,通过tiling参数传递给Kernel端。如果当前shape不满足regbase的硬件要求(如对齐约束),`isRegBase`会被置为`false`,此时三个算法分支均不会被选中,算子回退到通用(非regbase)路径。 | 112 | +`isRegBase`是一个由Host端根据输入shape和数据类型计算出的布尔值,通过tiling参数传递给Kernel端。如果当前shape不满足regbase的硬件要求(如对齐约束),`isRegBase`会被置为`false`,此时四个算法分支均不会被选中,算子回退到通用(非regbase)路径。 |
| 112 | 113 | ||
| 113 | **为什么Welford和TwoPass都要求isRegBase == true?**因为并行Welford和二分累加算法需要频繁访问M2累加器、临时缓冲区等,regbase模式能保证这些访问以最高效的方式进行。当前不存在不需要regbase的Welford/TwoPass降级路径 —— 如果不满足regbase条件,算子会回退到其他LayerNorm实现(如V3或通用路径)。 | 114 | **为什么Welford和TwoPass都要求isRegBase == true?**因为并行Welford和二分累加算法需要频繁访问M2累加器、临时缓冲区等,regbase模式能保证这些访问以最高效的方式进行。当前不存在不需要regbase的Welford/TwoPass降级路径 —— 如果不满足regbase条件,算子会回退到其他LayerNorm实现(如V3或通用路径)。 |
| 114 | 115 | ||
| @@ -195,19 +196,22 @@ y = gamma * ((x - mean) * rstd) + beta // 归一化 + 仿射变换 | |||
| 195 | 196 | ||
| 196 | ## 2. 总体架构 | 197 | ## 2. 总体架构 |
| 197 | 198 | ||
| 198 | -LayerNormV4在950(arch35)平台上有三个算法分支,通过**TILING_KEY**在编译期动态选择: | 199 | +LayerNormV4在950(arch35)平台上有四个算法分支,通过**TILING_KEY**在编译期动态选择: |
| 199 | 200 | ||
| 200 | | TILING_KEY | 算法 | 模板优先级 | 适用场景 | | 201 | | TILING_KEY | 算法 | 模板优先级 | 适用场景 | |
| 201 | |-----------|------|--------|---------| | 202 | |-----------|------|--------|---------| |
| 202 | -| 400-422 | Parallel Welford | 最高(4000) | 通用场景,单遍扫描 | | 203 | +| 800-822 | SplitR 跨核分段归约 | 最高(140,最先尝试) | 小A大R:R≥65536 且 A≤AIV核数/4,每核扫描连续片段后经workspace交换、一次全核同步与固定顺序合并 | |
| 204 | +| 400-422 | Parallel Welford | 最低(4000,最后尝试) | 通用场景,单遍扫描 | | ||
| 203 | | 300-322 | Two-Pass + Binary累加 | 中(200) | 超长轴场景(R较大,Welford不满足UB约束时) | | 205 | | 300-322 | Two-Pass + Binary累加 | 中(200) | 超长轴场景(R较大,Welford不满足UB约束时) | |
| 204 | -| 500-522 | Two-Pass Perf | 低(150) | 中等长度轴,极致性能(额外要求rowAlign不超硬限制、colSize/rowAlign不触发联立阈值) | | 206 | +| 500-522 | Two-Pass Perf | 次之(150) | 中等长度轴,极致性能(额外要求rowAlign不超硬限制、colSize/rowAlign不触发联立阈值) | |
| 205 | 207 | ||
| 206 | -**分派逻辑**(入口代码位于`op_kernel/layer_norm_v4_apt.h:80-143`): | 208 | +**分派逻辑**(入口代码位于`op_kernel/layer_norm_v4_apt.h:104-184`): |
| 207 | 209 | ||
| 208 | ```cpp | 210 | ```cpp |
| 209 | __aicore__ inline void layer_norm_impl(...) { | 211 | __aicore__ inline void layer_norm_impl(...) { |
| 210 | - if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) { | 212 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT)) { |
| 213 | + RegbaseSplitRImpl<float, float>(...); | ||
| 214 | + } else if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) { | ||
| 211 | RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...); | 215 | RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...); |
| 212 | } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) { | 216 | } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) { |
| 213 | RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...); | 217 | RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...); |
| @@ -217,7 +221,7 @@ __aicore__ inline void layer_norm_impl(...) { | |||
| 217 | } | 221 | } |
| 218 | ``` | 222 | ``` |
| 219 | 223 | ||
| 220 | -`TILING_KEY_IS()`宏在编译期展开为常量比较,三个分支中只有匹配的一个会被保留,其余在编译优化阶段被删除,无运行时开销。宏的参数命名规则参见 [前置知识0.3.3](#033-tiling_key--算法选择键)。 | 224 | +`TILING_KEY_IS()`宏在编译期展开为常量比较,四个分支中只有匹配的一个会被保留,其余在编译优化阶段被删除,无运行时开销。宏的参数命名规则参见 [前置知识0.3.3](#033-tiling_key--算法选择键)。 |
| 221 | 225 | ||
| 222 | --- | 226 | --- |
| 223 | 227 | ||
| @@ -409,7 +413,7 @@ Level 2: (s0..s3) (s4..s7) ← 仍相近 | |||
| 409 | Level 3: 最终总和(s0..s7) | 413 | Level 3: 最终总和(s0..s7) |
| 410 | ``` | 414 | ``` |
| 411 | 415 | ||
| 412 | -**TwoPass的空间与性能代价:**两遍扫描意味着数据从Global Memory搬入UB至少3次(第1遍算均值、第2遍算方差、归一化再1次),相比Welford的2次多了约50% 的带宽开销。这是TwoPass的优先级低于Welford的核心理由。但它的优势在于**中间状态极紧凑** —— 仅需存放二分累加树的临时缓冲区,不依赖每行的M2累加器。 | 416 | +**TwoPass的空间与性能代价:**两遍扫描意味着数据从Global Memory搬入UB至少3次(第1遍算均值、第2遍算方差、归一化再1次),相比Welford的2次多了约50% 的带宽开销。额外的带宽开销是模板准入与选择时权衡的因素之一。但它的优势在于**中间状态极紧凑** —— 仅需存放二分累加树的临时缓冲区,不依赖每行的M2累加器。 |
| 413 | 417 | ||
| 414 | --- | 418 | --- |
| 415 | 419 | ||
| @@ -522,23 +526,24 @@ CANN框架在Host端按照注册的优先级顺序依次尝试匹配Tiling模板 | |||
| 522 | 526 | ||
| 523 | | 优先级 | 模板 | 文件 | | 527 | | 优先级 | 模板 | 文件 | |
| 524 | |-------|------|------| | 528 | |-------|------|------| |
| 529 | +| 140 | LayerNormV4SplitRTiling | `op_host/layer_norm_v4_split_r_tiling.cpp` | | ||
| 525 | | 4000 | LayerNormV4WelfordTiling | `op_host/layer_norm_v4_welford_tiling.cpp` | | 530 | | 4000 | LayerNormV4WelfordTiling | `op_host/layer_norm_v4_welford_tiling.cpp` | |
| 526 | | 200 | LayerNormV4RegBaseTwoPassTiling | `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | | 531 | | 200 | LayerNormV4RegBaseTwoPassTiling | `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | |
| 527 | | 150 | LayerNormV4RegBaseTwoPassPerfTiling | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | | 532 | | 150 | LayerNormV4RegBaseTwoPassPerfTiling | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | |
| 528 | 533 | ||
| 529 | -优先级4000/200/150是模板**匹配优先级**(越高越先尝试),与TILING_KEY的范围值(400/300/500)是两套独立编号。框架先按优先级找模板,模板再打出对应的TILING_KEY。 | 534 | +模板按优先级数值从小到大尝试:SplitR(140)→TwoPassPerf(150)→TwoPass(200)→Welford(4000)。模板优先级决定匹配顺序,TILING_KEY 标识最终选中的算法与类型组合。 |
| 530 | 535 | ||
| 531 | -**选择逻辑:** | 536 | +SplitR 使用公共 `isRegBase` 平台判断,不额外限制确定性等级。准入形状为 `1 ≤ A ≤ floor(AIV核数/4)`、`65536 ≤ R ≤ INT32_MAX`;64个AIV核时即 `A ≤ 16`。每行分配 `min(16, floor(AIV核数/A), floor(R/16384))` 个核,至少2个核。其他形状继续按优先级尝试原模板。 |
| 532 | 537 | ||
| 533 | -1. **Welford优先(4000)**:当`isRegBase == true`且R不超过UB容量限制时被选中。Welford是首选的默认算法 —— 单遍扫描,精度最优,性价比最高 | 538 | +每核以 Welford 计算连续片段的均值和总体方差,写入独立的 mean、variance workspace 平面;`SyncAll()` 后通过 `DataCopyPad<float, PaddingMode::Compact>` 将每核记录中的首个 float 紧凑搬入 UB。两个平面各占一个向量,空间系数2表示两个统计量,并非双缓冲。最多16段的统计量使用掩码向量归约,不再需要通用 BatchNorm 的多级归约暂存。 |
| 534 | -2. **TwoPass备选(200)**:在Welford不可用时生效。Welford不可用的典型场景是R过大导致M2累加器数组超出UB容量。此时TwoPass的紧凑中间状态使其成为唯一可行的regbase选项 | 539 | + |
| 535 | -3. **TwoPassPerf降级(150)**:优先级最低。在其适用范围内(中等轴长+满足附加约束),通过(x-mean) 复用和行配对优化,性能在适用shape下可达到更优 | 540 | +令 `w_s = n_s / R`,以首段均值为参考计算 `mean = mean_0 + sum((mean_s - mean_0) * w_s)`,再计算 `variance = sum(variance_s * w_s + (mean_s - mean)^2 * w_s)`。权重先归一化再乘方差,避免 `variance_s * n_s` 的中间值溢出;平方项仍受 FP32 有限范围限制。每核归一化自身片段,仅首段输出 mean/rstd。 |
| 536 | 541 | ||
| 537 | --- | 542 | --- |
| 538 | 543 | ||
| 539 | ### 5.2 IsCapable() 检查条件详解 | 544 | ### 5.2 IsCapable() 检查条件详解 |
| 540 | 545 | ||
| 541 | -每个模板都有一个`IsCapable()`方法,在Host端运行时根据实际shape判断该模板是否可用。下表完整解析三个算法的Capable条件及其设计意图: | 546 | +每个模板都有一个`IsCapable()`方法,在Host端运行时根据实际shape判断该模板是否可用。下表列出原有三个算法的Capable条件;SplitR准入见5.1: |
| 542 | 547 | ||
| 543 | | 算法 | IsCapable条件 | 条件解释与设计意图 | | 548 | | 算法 | IsCapable条件 | 条件解释与设计意图 | |
| 544 | |------|---------------|------------------| | 549 | |------|---------------|------------------| |
| @@ -548,9 +553,9 @@ CANN框架在Host端按照注册的优先级顺序依次尝试匹配Tiling模板 | |||
| 548 | 553 | ||
| 549 | **TwoPassPerf的附加条件解释:** | 554 | **TwoPassPerf的附加条件解释:** |
| 550 | 555 | ||
| 551 | -- **rowAlign不超出硬限制**:`IsCapable()`的第一条几何判断(约第63行)为`rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`时返回false,即约`4 × VL_F32²`这一量级(`LN_NUM_TWO = 2`,代入后为`2 × 64 × 64 × 2 = 16384`)。这意味着rowAlign存在一个与`VL_F32²`相关的硬上限,超出则不可用。 | 556 | +- **rowAlign不超出硬限制**:`IsCapable()`的第一条几何判断(`op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp:70-73`)为`rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`时返回false,即约`4 × VL_F32²`这一量级(`LN_NUM_TWO = 2`,代入后为`2 × 64 × 64 × 2 = 16384`)。这意味着rowAlign存在一个与`VL_F32²`相关的硬上限,超出则不可用。 |
| 552 | 557 | ||
| 553 | -- **colSize/rowAlign不触发联立阈值**:第二条判断(约第68行)为`colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`时返回false。其中两个阈值为静态常量(= 4096和8192),分别对应colSize和rowAlign的上限。两条件需同时满足才触发拒绝 —— 这避免了单独一个维度偏大但另一个维度较小时误判不可用。 | 558 | +- **colSize/rowAlign不触发联立阈值**:第二条判断(第76-78行)为`colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`时返回false。其中两个阈值为静态常量(= 4096和8192),分别对应colSize和rowAlign的上限。两条件需同时满足才触发拒绝 —— 这避免了单独一个维度偏大但另一个维度较小时误判不可用。 |
| 554 | 559 | ||
| 555 | - **满足UB容量约束**:通过`CanFitInBuffer(1)`校验tile大小 + (x-mean) 差值缓冲区 + 其他临时空间总和不超过UB容量。 | 560 | - **满足UB容量约束**:通过`CanFitInBuffer(1)`校验tile大小 + (x-mean) 差值缓冲区 + 其他临时空间总和不超过UB容量。 |
| 556 | 561 | ||
| @@ -562,7 +567,7 @@ TwoPass的IsCapable条件: isRegBase + UB约束 | |||
| 562 | TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB约束 ← 最严格 | 567 | TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB约束 ← 最严格 |
| 563 | ``` | 568 | ``` |
| 564 | 569 | ||
| 565 | -框架从Welford(最宽松)开始检查,若满足则直接选中;不满足才逐级尝试更严格的分支。这与模板优先级(4000→200→150)的设计完全一致。 | 570 | +以上条件宽松程度不代表匹配顺序。框架按 SplitR→TwoPassPerf→TwoPass→Welford 的顺序检查,首个成功模板负责生成 tiling。 |
| 566 | 571 | ||
| 567 | ### 5.3 三维权衡分析:精度、性能与适用性 | 572 | ### 5.3 三维权衡分析:精度、性能与适用性 |
| 568 | 573 | ||
| @@ -574,7 +579,7 @@ TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB | |||
| 574 | | **TwoPass** | 超长轴适应性 | 多一遍全局数据搬运(3次vs 2次) | R小时性能损失明显,无精度优势 | | 579 | | **TwoPass** | 超长轴适应性 | 多一遍全局数据搬运(3次vs 2次) | R小时性能损失明显,无精度优势 | |
| 575 | | **TwoPassPerf** | 极致性能 + 极好精度 | rowAlign和colSize/rowAlign受联立阈值限制 | shape越界时不可用,回退到较慢分支 | | 580 | | **TwoPassPerf** | 极致性能 + 极好精度 | rowAlign和colSize/rowAlign受联立阈值限制 | shape越界时不可用,回退到较慢分支 | |
| 576 | 581 | ||
| 577 | -**设计哲学:**三套算法不是互相替代的关系,而是对"**shape空间**"的一个覆盖性划分。框架通过优先级(4000→200→150)对常见case做最优配置,通过IsCapable条件在极端case下做兜底保证。 | 582 | +**设计哲学:**三套算法不是互相替代的关系,而是对"**shape空间**"的一个覆盖性划分。框架通过优先级(140→150→200→4000)对常见case做最优配置,通过IsCapable条件在极端case下做兜底保证。 |
| 578 | 583 | ||
| 579 | 对于软件领域的读者来说,这种"多算法覆盖shape空间"的模式类似于: | 584 | 对于软件领域的读者来说,这种"多算法覆盖shape空间"的模式类似于: |
| 580 | 585 | ||
| @@ -596,7 +601,7 @@ TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB | |||
| 596 | | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Host CPU | TwoPass Perf tiling(含IsCapable中的A/R阈值定义) | | 601 | | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Host CPU | TwoPass Perf tiling(含IsCapable中的A/R阈值定义) | |
| 597 | | `op_kernel/layer_norm_v4_apt.h` | AI Core(设备端) | 950平台kernel入口和TILING_KEY分派 | | 602 | | `op_kernel/layer_norm_v4_apt.h` | AI Core(设备端) | 950平台kernel入口和TILING_KEY分派 | |
| 598 | | `op_host/layer_norm_v4_tiling.h` | Host CPU | Tiling数据结构定义(各模板共享) | | 603 | | `op_host/layer_norm_v4_tiling.h` | Host CPU | Tiling数据结构定义(各模板共享) | |
| 599 | -| `op_host/layer_norm_v4_def.cpp:191` | Host CPU | 950平台配置,注册为`layer_norm_v4_apt` | | 604 | +| `op_host/layer_norm_v4_def.cpp:171` | Host CPU | 950平台配置,注册为`layer_norm_v4_apt` | |
| 600 | 605 | ||
| 601 | 代码分层小结: | 606 | 代码分层小结: |
| 602 | 607 | ||
| @@ -612,7 +617,7 @@ LayerNormV4在Ascend 950平台上的精度优化体现了两个层面的设计 | |||
| 612 | 617 | ||
| 613 | - **算法层面**:Parallel Welford通过在线递推更新方式将中心化操作融入每一步,从根源上避免了朴素方差公式中`E[X²] - (E[X])²`大数相减的灾难性抵消(catastrophic cancellation);超长轴二分累加通过树状归约替代链式加法,使每一步相加的两个操作数量级相近,将累加舍入误差从O(n) 降至O(log n) | 618 | - **算法层面**:Parallel Welford通过在线递推更新方式将中心化操作融入每一步,从根源上避免了朴素方差公式中`E[X²] - (E[X])²`大数相减的灾难性抵消(catastrophic cancellation);超长轴二分累加通过树状归约替代链式加法,使每一步相加的两个操作数量级相近,将累加舍入误差从O(n) 降至O(log n) |
| 614 | 619 | ||
| 615 | -- **工程层面**:三套算法通过TILING_KEY机制在编译期动态选择,以模板优先级(4000/200/150)和IsCapable条件构成两级筛选,不同算法分别追求精度(Welford)、极端适应性(TwoPass)、极致性能(TwoPassPerf),共同覆盖了从短轴到超长轴的完整shape空间;UB空间的精细化管理(double buffer、tile大小对齐、临时缓冲区容量校验)确保了数据搬运与计算在AI Core上的高效流水 | 620 | +- **工程层面**:三套算法通过TILING_KEY机制在编译期动态选择,以模板优先级(140/150/200/4000)和IsCapable条件构成两级筛选,不同算法分别追求精度(Welford)、极端适应性(TwoPass)、极致性能(TwoPassPerf),共同覆盖了从短轴到超长轴的完整shape空间;UB空间的精细化管理(double buffer、tile大小对齐、临时缓冲区容量校验)确保了数据搬运与计算在AI Core上的高效流水 |
| 616 | 621 | ||
| 617 | 这两种算法思想不仅适用于LayerNormV4,对于所有需要沿超长轴做归约的Norm类算子(如 **RMSNorm**、**GroupNorm**、**InstanceNorm**等),以及更广义的高精度浮点归约场景,都具有普适的参考价值。 | 622 | 这两种算法思想不仅适用于LayerNormV4,对于所有需要沿超长轴做归约的Norm类算子(如 **RMSNorm**、**GroupNorm**、**InstanceNorm**等),以及更广义的高精度浮点归约场景,都具有普适的参考价值。 |
| 618 | 623 | ||
| @@ -81,6 +81,7 @@ Similarly, `LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT` indicates the Two-Pass (binary ac | |||
| 81 | 81 | ||
| 82 | | TILING_KEY range | Algorithm | Meaning of the base | | 82 | | TILING_KEY range | Algorithm | Meaning of the base | |
| 83 | |----------------|------|---------| | 83 | |----------------|------|---------| |
| 84 | +| 800-822 | SplitR | Base 800 = starting number of the SplitR algorithm family | | ||
| 84 | | 400-422 | Parallel Welford | Base 400 = starting number of the Welford algorithm family | | 85 | | 400-422 | Parallel Welford | Base 400 = starting number of the Welford algorithm family | |
| 85 | | 300-322 | Two-Pass + Binary accumulation | Base 300 = starting number of the TwoPass algorithm family | | 86 | | 300-322 | Two-Pass + Binary accumulation | Base 300 = starting number of the TwoPass algorithm family | |
| 86 | | 500-522 | Two-Pass Perf (optimized version) | Base 500 = starting number of the TwoPassPerf algorithm family | | 87 | | 500-522 | Two-Pass Perf (optimized version) | Base 500 = starting number of the TwoPassPerf algorithm family | |
| @@ -98,7 +99,7 @@ This lets the compiler eliminate mismatched branches at compile time and remove | |||
| 98 | 99 | ||
| 99 | **Relationship between TILING_KEY and Tiling template priority:** | 100 | **Relationship between TILING_KEY and Tiling template priority:** |
| 100 | 101 | ||
| 101 | -When a Tiling template is registered, it carries a priority value (for example, Welford is 4000, TwoPass is 200, and TwoPassPerf is 150). The framework tries matching from the highest priority to the lowest: it first checks whether Welford (4000) satisfies `IsCapable()`; if not, it tries TwoPass (200); if that still fails, it tries TwoPassPerf (150). This priority system and the TILING_KEY range values (400/300/500) are two independent numbering systems — the former controls the template matching order, and the latter is the algorithm identifier. | 102 | +When a Tiling template is registered, it carries a priority value (for example, SplitR is 140, TwoPassPerf is 150, TwoPass is 200, and Welford is 4000). The framework tries matching in ascending priority order: it first checks whether SplitR (140) satisfies `IsCapable()`; if not, it tries TwoPassPerf (150), then TwoPass (200), and finally Welford (4000). This priority system and the TILING_KEY range values (400/300/500/800) are two independent numbering systems — the former controls the template matching order, and the latter is the algorithm identifier. |
| 102 | 103 | ||
| 103 | #### 0.3.4 regbase (Register Base Addressing Mode) | 104 | #### 0.3.4 regbase (Register Base Addressing Mode) |
| 104 | 105 | ||
| @@ -193,21 +194,24 @@ On the Ascend 950 platform, the LayerNormV4 operator uses two core algorithms, * | |||
| 193 | 194 | ||
| 194 | --- | 195 | --- |
| 195 | 196 | ||
| 196 | -## 2. Overall Architecture | 197 | +## 2. Architecture |
| 197 | 198 | ||
| 198 | -LayerNormV4 has three algorithm branches on the 950 (arch35) platform, dynamically selected at compile time through **TILING_KEY**: | 199 | +LayerNormV4 has four algorithm branches on the 950 (arch35) platform, dynamically selected at compile time through **TILING_KEY**: |
| 199 | 200 | ||
| 200 | | TILING_KEY | Algorithm | Template Priority | Applicable Scenario | | 201 | | TILING_KEY | Algorithm | Template Priority | Applicable Scenario | |
| 201 | |-----------|------|--------|---------| | 202 | |-----------|------|--------|---------| |
| 202 | -| 400-422 | Parallel Welford | Highest (4000) | General scenarios, single-pass scan | | 203 | +| 800-822 | SplitR cross-core segmented reduction | Highest (140, tried first) | Small A, large R: R≥65536 and A≤AIV cores/4; each core scans a contiguous segment, then exchanges via workspace, one whole-core synchronization, and a fixed-order merge | |
| 204 | +| 400-422 | Parallel Welford | Lowest (4000, tried last) | General scenarios, single-pass scan | | ||
| 203 | | 300-322 | Two-Pass + Binary accumulation | Medium (200) | Extra-long axis scenarios (R is large and Welford does not satisfy the UB constraint) | | 205 | | 300-322 | Two-Pass + Binary accumulation | Medium (200) | Extra-long axis scenarios (R is large and Welford does not satisfy the UB constraint) | |
| 204 | -| 500-522 | Two-Pass Perf | Low (150) | Medium-length axes, extreme performance (additional requirements: rowAlign does not exceed the hard limit, and colSize/rowAlign does not trigger the joint threshold) | | 206 | +| 500-522 | Two-Pass Perf | Next (150) | Medium-length axes, extreme performance (additional requirements: rowAlign does not exceed the hard limit, and colSize/rowAlign does not trigger the joint threshold) | |
| 205 | 207 | ||
| 206 | -**Dispatch logic** (entry code in `op_kernel/layer_norm_v4_apt.h:80-143`): | 208 | +**Dispatch logic** (entry code in `op_kernel/layer_norm_v4_apt.h:104-184`): |
| 207 | 209 | ||
| 208 | ```cpp | 210 | ```cpp |
| 209 | __aicore__ inline void layer_norm_impl(...) { | 211 | __aicore__ inline void layer_norm_impl(...) { |
| 210 | - if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) { | 212 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT)) { |
| 213 | + RegbaseSplitRImpl<float, float>(...); | ||
| 214 | + } else if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) { | ||
| 211 | RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...); | 215 | RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...); |
| 212 | } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) { | 216 | } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) { |
| 213 | RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...); | 217 | RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...); |
| @@ -217,9 +221,9 @@ __aicore__ inline void layer_norm_impl(...) { | |||
| 217 | } | 221 | } |
| 218 | ``` | 222 | ``` |
| 219 | 223 | ||
| 220 | -The `TILING_KEY_IS()` macro expands into a constant comparison at compile time. Only the matching one of the three branches is retained, and the others are removed during compile-time optimization, with no runtime overhead. For the naming rules of macro parameters, see [Prerequisite Knowledge 0.3.3](#033-tiling_key--algorithm-selection-key). | 224 | +The `TILING_KEY_IS()` macro expands into a constant comparison at compile time. Only the matching one of the four branches is retained, and the others are removed during compile-time optimization, with no runtime overhead. For the naming rules of macro parameters, see [Prerequisite Knowledge 0.3.3](#033-tiling_key--algorithm-selection-key). |
| 221 | 225 | ||
| 222 | ---- | 226 | +Welford and SplitR share buffer initialization, local statistics, normalization, and cache management in `LayerNormV4WelfordBase`. The whole-row Welford class controls row iteration. SplitR distributes one row across multiple cores and merges segment statistics. Unmatched SplitR shapes continue through Two-Pass Perf, Two-Pass, and Welford template selection. The selected key also encodes the supported input and parameter data types. |
| 223 | 227 | ||
| 224 | ## 3. Parallel Welford Algorithm | 228 | ## 3. Parallel Welford Algorithm |
| 225 | 229 | ||
| @@ -409,7 +413,7 @@ Level 2: (s0..s3) (s4..s7) ← still similar | |||
| 409 | Level 3: final sum (s0..s7) | 413 | Level 3: final sum (s0..s7) |
| 410 | ``` | 414 | ``` |
| 411 | 415 | ||
| 412 | -**Space and performance cost of TwoPass:** Two passes mean the data is moved from Global Memory into UB at least 3 times (the first pass computes the mean, the second pass computes the variance, and normalization takes 1 more time), about 50% more bandwidth overhead than the 2 times of Welford. This is the core reason why TwoPass has a lower priority than Welford. But its advantage is an **extremely compact intermediate state** — it only needs to store the temporary buffer of the binary accumulation tree and does not depend on per-row M2 accumulators. | 416 | +**Space and performance cost of TwoPass:** Two passes mean the data is moved from Global Memory into UB at least 3 times (the first pass computes the mean, the second pass computes the variance, and normalization takes 1 more time), about 50% more bandwidth overhead than the 2 times of Welford. Its additional bandwidth cost is one tradeoff considered by template admission and selection. But its advantage is an **extremely compact intermediate state** — it only needs to store the temporary buffer of the binary accumulation tree and does not depend on per-row M2 accumulators. |
| 413 | 417 | ||
| 414 | --- | 418 | --- |
| 415 | 419 | ||
| @@ -514,31 +518,45 @@ The `LAST_LOOP_NUMS` template parameter controls the merge strategy of the **las | |||
| 514 | 518 | ||
| 515 | --- | 519 | --- |
| 516 | 520 | ||
| 517 | -## 5. Algorithm Selection Strategy | 521 | +## 5. Algorithm Selection |
| 518 | 522 | ||
| 519 | -### 5.1 Tiling Template Registration Priorities | 523 | +### 5.1 Priority and SplitR admission |
| 520 | 524 | ||
| 521 | -On the Host side, the CANN framework tries to match Tiling templates one by one in the registered priority order: | 525 | +Templates are attempted in this order: |
| 522 | 526 | ||
| 523 | -| Priority | Template | File | | 527 | +1. SplitR: priority 140. |
| 524 | -|-------|------|------| | 528 | +2. Two-Pass Perf: priority 150. |
| 525 | -| 4000 | LayerNormV4WelfordTiling | `op_host/layer_norm_v4_welford_tiling.cpp` | | 529 | +3. Two-Pass: priority 200. |
| 526 | -| 200 | LayerNormV4RegBaseTwoPassTiling | `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | | 530 | +4. Welford: priority 4000. |
| 527 | -| 150 | LayerNormV4RegBaseTwoPassPerfTiling | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | | ||
| 528 | 531 | ||
| 529 | -The priorities 4000/200/150 are the **matching priorities** of templates (higher values are tried first) and are independent of the TILING_KEY range values (400/300/500). The framework first finds a template by priority, and the template then produces the corresponding TILING_KEY. | 532 | +SplitR uses the shared `isRegBase` platform check and excludes the V3 path. It does not add a deterministic-level restriction. Its shape conditions are: |
| 530 | 533 | ||
| 531 | -**Selection logic:** | 534 | +```text |
| 535 | +1 <= A <= floor(AIV_core_count / 4) | ||
| 536 | +65536 <= R <= INT32_MAX | ||
| 537 | +splitR = min(16, floor(AIV_core_count / A), floor(R / 16384)) | ||
| 538 | +splitR >= 2 | ||
| 539 | +``` | ||
| 532 | 540 | ||
| 533 | -1. **Welford first (4000)**: selected when `isRegBase == true` and R does not exceed the UB capacity limit. Welford is the preferred default algorithm — single-pass scan, best precision, and best cost-effectiveness | 541 | +For 64 AIV cores, A must be at most 16. Segment boundaries are aligned, and the last segment uses its actual element count. |
| 534 | -2. **TwoPass as fallback (200)**: takes effect when Welford is unavailable. The typical scenario where Welford is unavailable is that R is so large that the M2 accumulator array exceeds the UB capacity. At this point, the compact intermediate state of TwoPass makes it the only feasible regbase option | ||
| 535 | -3. **TwoPassPerf downgrade (150)**: lowest priority. Within its applicable range (medium axis length + additional constraints satisfied), through (x-mean) reuse and row pairing optimizations, its performance can be better for the applicable shapes | ||
| 536 | 542 | ||
| 537 | ---- | 543 | +Each core computes a segment mean and population variance and publishes them into separate workspace planes. Each GM record occupies one UB block. After `SyncAll`, `DataCopyPad<float, PaddingMode::Compact>` gathers the first float of each record into adjacent UB lanes. Each of the two planes is rounded up to a whole number of vectors; the factor two represents mean and variance, independently of double buffering. |
| 544 | + | ||
| 545 | +At most sixteen segment statistics fit in one vector. The merge builds weighted contributions in vector-sized chunks and invokes the common `NormCommon::NormCommonRegbase::CalculateReduceSum` interface: | ||
| 546 | + | ||
| 547 | +```text | ||
| 548 | +weight_s = count_s / R | ||
| 549 | +mean = mean_0 + sum((mean_s - mean_0) * weight_s) | ||
| 550 | +variance = sum(variance_s * weight_s + (mean_s - mean)^2 * weight_s) | ||
| 551 | +``` | ||
| 552 | + | ||
| 553 | +Centering the mean around the first segment retains the large-offset precision fix. Normalizing counts before multiplying variance avoids the intermediate `variance_s * count_s` overflow. The squared mean-difference term remains subject to FP32 range limits. No general multi-level BatchNorm reduction scratch is required. | ||
| 554 | + | ||
| 555 | +Each core normalizes its own input segment. Only the first segment writes mean/rstd. Workspace includes the system synchronization allocation plus the two statistic planes. Host UB budgeting reserves both vector-aligned compact planes before selecting tile length. | ||
| 538 | 556 | ||
| 539 | ### 5.2 Detailed IsCapable() Check Conditions | 557 | ### 5.2 Detailed IsCapable() Check Conditions |
| 540 | 558 | ||
| 541 | -Each template has an `IsCapable()` method that determines at runtime on the Host side whether the template is available for the actual shape. The following table fully analyzes the Capable conditions of the three algorithms and their design intent: | 559 | +Each template has an `IsCapable()` method that determines at runtime on the Host side whether the template is available for the actual shape. The following table lists the Capable conditions of the original three algorithms; SplitR admission is described in 5.1: |
| 542 | 560 | ||
| 543 | | Algorithm | IsCapable Condition | Condition Explanation and Design Intent | | 561 | | Algorithm | IsCapable Condition | Condition Explanation and Design Intent | |
| 544 | |------|---------------|------------------| | 562 | |------|---------------|------------------| |
| @@ -548,21 +566,20 @@ Each template has an `IsCapable()` method that determines at runtime on the Host | |||
| 548 | 566 | ||
| 549 | **Explanation of the additional conditions of TwoPassPerf:** | 567 | **Explanation of the additional conditions of TwoPassPerf:** |
| 550 | 568 | ||
| 551 | -- **rowAlign does not exceed the hard limit**: the first geometric judgment of `IsCapable()` (about line 63) returns false when `rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`, that is, the magnitude of about `4 × VL_F32²` (`LN_NUM_TWO = 2`, substituting gives `2 × 64 × 64 × 2 = 16384`). This means rowAlign has a hard upper limit related to `VL_F32²`; beyond it, the template is unavailable. | 569 | +- **rowAlign does not exceed the hard limit**: the first geometric judgment of `IsCapable()` (`op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp:70-73`) returns false when `rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`, that is, the magnitude of about `4 × VL_F32²` (`LN_NUM_TWO = 2`, substituting gives `2 × 64 × 64 × 2 = 16384`). This means rowAlign has a hard upper limit related to `VL_F32²`; beyond it, the template is unavailable. |
| 552 | 570 | ||
| 553 | -- **colSize/rowAlign does not trigger the joint threshold**: the second judgment (about line 68) returns false when `colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`. The two thresholds are static constants (= 4096 and 8192), corresponding to the upper limits of colSize and rowAlign respectively. Both conditions must be satisfied simultaneously to trigger rejection — this avoids misjudging unavailability when only one dimension is too large while the other is small. | 571 | +- **colSize/rowAlign does not trigger the joint threshold**: the second judgment (lines 76-78) returns false when `colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`. The two thresholds are static constants (= 4096 and 8192), corresponding to the upper limits of colSize and rowAlign respectively. Both conditions must be satisfied simultaneously to trigger rejection — this avoids misjudging unavailability when only one dimension is too large while the other is small. |
| 554 | 572 | ||
| 555 | - **UB capacity constraint satisfied**: `CanFitInBuffer(1)` validates that the total of the tile size + the (x-mean) difference buffer + other temporary space does not exceed the UB capacity. | 573 | - **UB capacity constraint satisfied**: `CanFitInBuffer(1)` validates that the total of the tile size + the (x-mean) difference buffer + other temporary space does not exceed the UB capacity. |
| 556 | 574 | ||
| 557 | **Logical relationship between the conditions (from strict to relaxed):** | 575 | **Logical relationship between the conditions (from strict to relaxed):** |
| 558 | - | ||
| 559 | ```Cpp | 576 | ```Cpp |
| 560 | Welford IsCapable condition: isRegBase == true ← most relaxed | 577 | Welford IsCapable condition: isRegBase == true ← most relaxed |
| 561 | TwoPass IsCapable condition: isRegBase + UB constraint ← medium | 578 | TwoPass IsCapable condition: isRegBase + UB constraint ← medium |
| 562 | TwoPassPerf IsCapable condition: isRegBase + rowAlign hard limit + joint threshold + UB constraint ← strictest | 579 | TwoPassPerf IsCapable condition: isRegBase + rowAlign hard limit + joint threshold + UB constraint ← strictest |
| 563 | ``` | 580 | ``` |
| 564 | 581 | ||
| 565 | -The framework starts checking from Welford (most relaxed); if satisfied, it selects it directly; only if not satisfied does it try stricter branches level by level. This fully matches the design of template priorities (4000→200→150). | 582 | +The strictness order above does not represent the matching order. The framework checks in the order SplitR → TwoPassPerf → TwoPass → Welford, and the first successful template generates the tiling. |
| 566 | 583 | ||
| 567 | ### 5.3 Three-Way Trade-Off Analysis: Precision, Performance, and Applicability | 584 | ### 5.3 Three-Way Trade-Off Analysis: Precision, Performance, and Applicability |
| 568 | 585 | ||
| @@ -574,47 +591,33 @@ No algorithm can be optimal in all three dimensions at the same time. Understand | |||
| 574 | | **TwoPass** | Adaptability to extra-long axes | One extra global data transfer (3 times vs 2 times) | Obvious performance loss when R is small; no precision advantage | | 591 | | **TwoPass** | Adaptability to extra-long axes | One extra global data transfer (3 times vs 2 times) | Obvious performance loss when R is small; no precision advantage | |
| 575 | | **TwoPassPerf** | Extreme performance + excellent precision | rowAlign and colSize/rowAlign are limited by the joint threshold | Unavailable when the shape exceeds the bounds; falls back to a slower branch | | 592 | | **TwoPassPerf** | Extreme performance + excellent precision | rowAlign and colSize/rowAlign are limited by the joint threshold | Unavailable when the shape exceeds the bounds; falls back to a slower branch | |
| 576 | 593 | ||
| 577 | -**Design philosophy:** The three algorithms do not replace each other; instead, they form a covering partition of the "**shape space**". The framework uses priorities (4000→200→150) to configure common cases optimally, and uses IsCapable conditions to guarantee a fallback in extreme cases. | 594 | +**Design philosophy:** The three original algorithms do not replace each other; instead, they form a covering partition of the "**shape space**". The framework configures common cases optimally through priorities (140→150→200→4000), and IsCapable conditions guarantee a fallback in extreme cases. SplitR increases core utilization for small A and large R, adding workspace traffic and global synchronization. Shapes outside its threshold use the original templates. |
| 578 | 595 | ||
| 579 | For readers from the software field, this "multiple algorithms covering the shape space" pattern is similar to: | 596 | For readers from the software field, this "multiple algorithms covering the shape space" pattern is similar to: |
| 580 | 597 | ||
| 581 | - **Multi-level optimization passes of compilers** — different optimization levels suit different code characteristics | 598 | - **Multi-level optimization passes of compilers** — different optimization levels suit different code characteristics |
| 582 | -- **Multiple plan candidates of database query optimizers** — selecting the optimal execution plan based on a cost model | 599 | +- **Multi-plan candidates of database query optimizers** — the cost model selects the best execution plan |
| 583 | -- **Kernel selection of BLAS libraries** — MKL/OpenBLAS dynamically switches between different kernels based on matrix size | 600 | +- **Kernel selection in BLAS libraries** — MKL/OpenBLAS dynamically switch kernels based on matrix size |
| 584 | 601 | ||
| 585 | ---- | 602 | +## 6. Source Map |
| 586 | 603 | ||
| 587 | -## 6. Quick Reference of Code Structure | 604 | +| File | Responsibility | |
| 588 | - | 605 | +| --- | --- | |
| 589 | -| File | Execution Location | Content | | 606 | +| `op_host/layer_norm_v4_split_r_tiling.cpp` | SplitR admission, UB budget, workspace and scheduling | |
| 590 | -|------|---------|------| | 607 | +| `op_kernel/arch35/layer_norm_v4_split_r.h` | Segmentation, publication, compact gather, merge and normalization | |
| 591 | -| `op_kernel/arch35/layer_norm_v4_welford.h` | AI Core (device side) | Parallel Welford kernel implementation | | 608 | +| `op_kernel/arch35/layer_norm_v4_welford_base.h` | Shared Welford buffers and computation | |
| 592 | -| `op_host/layer_norm_v4_welford_tiling.cpp` | Host CPU | Welford tiling parameter computation (including IsCapable) | | 609 | +| `op_kernel/arch35/layer_norm_v4_welford.h` | Whole-row Welford execution | |
| 593 | -| `op_kernel/arch35/layer_norm_v4_two_pass.h` | AI Core (device side) | TwoPass + binary accumulation kernel implementation | | 610 | +| `op_host/layer_norm_v4_welford_tiling.cpp` | Welford tile selection | |
| 594 | -| `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | Host CPU | TwoPass tiling parameter computation (including IsCapable, DoOpTiling) | | 611 | +| `op_kernel/arch35/layer_norm_v4_two_pass.h` | Two-Pass computation | |
| 595 | -| `op_kernel/arch35/layer_norm_v4_two_pass_perf.h` | AI Core (device side) | TwoPass Perf optimized kernel | | 612 | +| `op_kernel/arch35/layer_norm_v4_two_pass_perf.h` | Optimized Two-Pass computation | |
| 596 | -| `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Host CPU | TwoPass Perf tiling (including A/R threshold definitions in IsCapable) | | 613 | +| `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | Two-Pass tiling | |
| 597 | -| `op_kernel/layer_norm_v4_apt.h` | AI Core (device side) | 950 platform kernel entry and TILING_KEY dispatch | | 614 | +| `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Two-Pass Perf tiling | |
| 598 | -| `op_host/layer_norm_v4_tiling.h` | Host CPU | Tiling data structure definitions (shared by templates) | | 615 | +| `op_kernel/layer_norm_v4_apt.h` | Kernel dispatch | |
| 599 | -| `op_host/layer_norm_v4_def.cpp:191` | Host CPU | 950 platform configuration, registered as `layer_norm_v4_apt` | | 616 | +| `op_host/layer_norm_v4_tiling.h` | Tiling data definitions | |
| 600 | - | ||
| 601 | -Summary of code layering: | ||
| 602 | - | ||
| 603 | -- The `.h` files under `op_kernel/arch35/` are **architecture-specific device-side kernel code** and run on the AI Core of Ascend 950 | ||
| 604 | -- The `.cpp`/`.h` files under `op_host/` are **Host-side control code** and run on the server CPU, responsible for parameter computation and scheduling | ||
| 605 | -- Developing an operator requires maintaining code on both sides — the Host side does "strategy computation", and the Kernel side does "strategy execution" | ||
| 606 | - | ||
| 607 | ---- | ||
| 608 | 617 | ||
| 609 | ## 7. Summary | 618 | ## 7. Summary |
| 610 | 619 | ||
| 611 | -The precision optimization of LayerNormV4 on the Ascend 950 platform reflects design ideas at two levels: | 620 | +Centered statistics, tree accumulation, and shape-based template selection address different numerical and performance constraints. SplitR extends this selection to small-A/large-R inputs while reusing the local Welford and normalization paths. Changes to reduction order and memory layout require precision and bounds regression on the resulting binary. |
| 612 | - | ||
| 613 | -- **Algorithm level**: Parallel Welford integrates centering into every step through online recurrence updates, avoiding at the root the catastrophic cancellation of subtracting large numbers in the naive variance formula `E[X²] - (E[X])²`; extra-long-axis binary accumulation replaces chained addition with tree-shaped reduction, making the two operands of each addition similar in magnitude, and reduces the accumulation rounding error from O(n) to O(log n) | ||
| 614 | - | ||
| 615 | -- **Engineering level**: the three algorithms are dynamically selected at compile time through the TILING_KEY mechanism, with template priorities (4000/200/150) and IsCapable conditions forming a two-level filter. The algorithms pursue precision (Welford), extreme adaptability (TwoPass), and extreme performance (TwoPassPerf) respectively, jointly covering the complete shape space from short axes to extra-long axes; fine-grained management of UB space (double buffer, tile size alignment, temporary buffer capacity validation) ensures efficient pipelining of data transfer and computation on the AI Core | ||
| 616 | - | ||
| 617 | -These two algorithm ideas apply not only to LayerNormV4 but also to all Norm operators that need to reduce along extra-long axes (such as **RMSNorm**, **GroupNorm**, and **InstanceNorm**), and more broadly to high-precision floating-point reduction scenarios. They have universal reference value. | ||
| 618 | 621 | ||
| 619 | --- | 622 | --- |
| 620 | 623 | ||
| @@ -0,0 +1,121 @@ | |||
| 1 | +/** | ||
| 2 | + * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | + * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | + * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | + * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | + * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | + * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | + */ | ||
| 10 | + | ||
| 11 | +/*! | ||
| 12 | + * \file layer_norm_v4_split_r_tiling.cpp | ||
| 13 | + * \brief | ||
| 14 | + */ | ||
| 15 | + | ||
| 16 | +#include <climits> | ||
| 17 | +#include "layer_norm_v4_tiling.h" | ||
| 18 | + | ||
| 19 | +namespace optiling { | ||
| 20 | +namespace { | ||
| 21 | +constexpr int64_t SPLIT_R_TEMPLATE = 8; | ||
| 22 | +// Conservative small-A/large-R dispatch policy, not hardware limits. | ||
| 23 | +// Keep enough work per split to amortize workspace traffic and SyncAll. | ||
| 24 | +constexpr int64_t SPLIT_R_MIN_ELEMENTS = 16384; | ||
| 25 | +constexpr int64_t SPLIT_R_MIN_ROW = 65536; | ||
| 26 | +constexpr int64_t SPLIT_R_MAX_CORES = 16; | ||
| 27 | +constexpr int64_t SPLIT_R_CORES_PER_ROW = 4; | ||
| 28 | +constexpr int64_t SPLIT_R_MIN_CORES = 2; | ||
| 29 | +constexpr uint64_t STATISTIC_PLANES = 2; | ||
| 30 | +} // namespace | ||
| 31 | + | ||
| 32 | +class LayerNormV4SplitRTiling : public LayerNormV4WelfordTiling { | ||
| 33 | +public: | ||
| 34 | + explicit LayerNormV4SplitRTiling(gert::TilingContext* context) : LayerNormV4WelfordTiling(context) {} | ||
| 35 | + | ||
| 36 | +protected: | ||
| 37 | + bool IsCapable() override | ||
| 38 | + { | ||
| 39 | + // A <= floor(AIV cores / 4), 65536 <= R <= INT32_MAX. | ||
| 40 | + // With 64 AIV cores this admits A <= 16. Each split has >= 16384 elements. | ||
| 41 | + if (!commonParams.isRegBase || commonParams.isV3) { | ||
| 42 | + return false; | ||
| 43 | + } | ||
| 44 | + if (commonParams.colSize == 0 || commonParams.colSize > commonParams.coreNum / SPLIT_R_CORES_PER_ROW) { | ||
| 45 | + return false; | ||
| 46 | + } | ||
| 47 | + splitR_ = std::min<int64_t>(commonParams.coreNum / commonParams.colSize, SPLIT_R_MAX_CORES); | ||
| 48 | + if (commonParams.rowSize < SPLIT_R_MIN_ROW || commonParams.rowSize > INT32_MAX) { | ||
| 49 | + return false; | ||
| 50 | + } | ||
| 51 | + splitR_ = std::min<int64_t>(splitR_, commonParams.rowSize / SPLIT_R_MIN_ELEMENTS); | ||
| 52 | + return splitR_ >= SPLIT_R_MIN_CORES; | ||
| 53 | + } | ||
| 54 | + | ||
| 55 | + // The UB deduction must wrap the base DoLibApiTiling call: the tileLength | ||
| 56 | + // selection loop (IsValidTileLength vs commonParams.ubSizePlatForm) runs | ||
| 57 | + // inside it, so the reserved partial-queue UB is excluded before | ||
| 58 | + // tileLength is chosen. The base DoOpTiling only derives the block | ||
| 59 | + // partitioning from M/coreNum and reads no UB budget, so it needs no | ||
| 60 | + // deduction and is inherited unchanged. | ||
| 61 | + ge::graphStatus DoLibApiTiling() override | ||
| 62 | + { | ||
| 63 | + // Separate mean/variance planes coexist with the Welford buffers. | ||
| 64 | + // Match the vector-aligned partial planes allocated by the kernel. | ||
| 65 | + uint64_t vectorBytes = Ops::Base::GetVRegSize(context_); | ||
| 66 | + uint64_t planeBytes = ((splitR_ * sizeof(float) + vectorBytes - 1) / vectorBytes) * vectorBytes; | ||
| 67 | + uint64_t reserved = STATISTIC_PLANES * planeBytes; | ||
| 68 | + if (commonParams.ubSizePlatForm <= reserved) { | ||
| 69 | + return ge::GRAPH_FAILED; | ||
| 70 | + } | ||
| 71 | + commonParams.ubSizePlatForm -= reserved; | ||
| 72 | + auto status = LayerNormV4WelfordTiling::DoLibApiTiling(); | ||
| 73 | + commonParams.ubSizePlatForm += reserved; | ||
| 74 | + return status; | ||
🟡 Medium Priority 建议:把 UB 预留从「仅包裹基类 DoLibApiTiling」改为「覆盖 DoOpTiling 与 DoLibApiTiling」:例如重写 DoOpTiling(),在其中先扣减 (splitR_+1)*PARTIAL_RECORD_BYTES 再调用基类 DoOpTiling/DoLibApiTiling,结束后恢复,或直接返回失败,避免基类 IsValidTileLength 用未扣减的 ubSizePlatForm 选出导致 UB 超额的 tileLength。 ![]() ![]() Incorrect? | |||
| 75 | + } | ||
| 76 | + | ||
| 77 | + uint64_t GetTilingKey() const override { return SPLIT_R_TEMPLATE * LN_TEMPLATE_KEY_WEIGHT + commonParams.dtypeKey; } | ||
| 78 | + | ||
| 79 | + ge::graphStatus PostTiling() override | ||
| 80 | + { | ||
| 81 | + splitTd_.set_M(td_.get_M()); | ||
| 82 | + splitTd_.set_N(td_.get_N()); | ||
| 83 | + splitTd_.set_rAlign(td_.get_rAlign()); | ||
| 84 | + splitTd_.set_numBlocks(td_.get_numBlocks()); | ||
| 85 | + splitTd_.set_mainBlockCount(td_.get_mainBlockCount()); | ||
| 86 | + splitTd_.set_mainBlockFactor(td_.get_mainBlockFactor()); | ||
| 87 | + splitTd_.set_tailBlockFactor(td_.get_tailBlockFactor()); | ||
| 88 | + splitTd_.set_tileLength(td_.get_tileLength()); | ||
| 89 | + splitTd_.set_welfordTempSize(td_.get_welfordTempSize()); | ||
| 90 | + splitTd_.set_welfordUpdateTimes(td_.get_welfordUpdateTimes()); | ||
| 91 | + splitTd_.set_welfordUpdateTail(td_.get_welfordUpdateTail()); | ||
| 92 | + splitTd_.set_nullptrGamma(td_.get_nullptrGamma()); | ||
| 93 | + splitTd_.set_nullptrBeta(td_.get_nullptrBeta()); | ||
| 94 | + splitTd_.set_apiTempBufferSize(td_.get_apiTempBufferSize()); | ||
| 95 | + splitTd_.set_epsilon(td_.get_epsilon()); | ||
| 96 | + splitTd_.set_splitR(splitR_); | ||
| 97 | + context_->SetBlockDim(commonParams.colSize * splitR_); | ||
| 98 | + auto* raw = context_->GetRawTilingData(); | ||
| 99 | + if (splitTd_.GetDataSize() > raw->GetCapacity()) { | ||
| 100 | + return ge::GRAPH_FAILED; | ||
| 101 | + } | ||
| 102 | + splitTd_.SaveToBuffer(raw->GetData(), raw->GetCapacity()); | ||
| 103 | + raw->SetDataSize(splitTd_.GetDataSize()); | ||
| 104 | + auto* sizes = context_->GetWorkspaceSizes(1); | ||
| 105 | + auto* info = context_->GetPlatformInfo(); | ||
| 106 | + OP_CHECK_NULL_WITH_CONTEXT(context_, sizes); | ||
| 107 | + OP_CHECK_NULL_WITH_CONTEXT(context_, info); | ||
| 108 | + platform_ascendc::PlatformAscendC platform(info); | ||
| 109 | + // Two statistic planes (mean and variance), independent of double buffering. | ||
| 110 | + sizes[0] = platform.GetLibApiWorkSpaceSize() + | ||
| 111 | + STATISTIC_PLANES * commonParams.colSize * splitR_ * Ops::Base::GetUbBlockSize(context_); | ||
| 112 | + return context_->SetScheduleMode(1U); | ||
| 113 | + } | ||
| 114 | + | ||
| 115 | +private: | ||
| 116 | + int64_t splitR_ = 1; | ||
| 117 | + LayerNormV4TilingDataSplitR splitTd_; | ||
| 118 | +}; | ||
| 119 | + | ||
| 120 | +REGISTER_OPS_TILING_TEMPLATE(LayerNormV4, LayerNormV4SplitRTiling, 140); | ||
| 121 | +} // namespace optiling | ||
| @@ -131,6 +131,31 @@ REGISTER_TILING_DATA_CLASS(LayerNormV4_411, LayerNormV4TilingDataWelford) | |||
| 131 | REGISTER_TILING_DATA_CLASS(LayerNormV4_420, LayerNormV4TilingDataWelford) | 131 | REGISTER_TILING_DATA_CLASS(LayerNormV4_420, LayerNormV4TilingDataWelford) |
| 132 | REGISTER_TILING_DATA_CLASS(LayerNormV4_422, LayerNormV4TilingDataWelford) | 132 | REGISTER_TILING_DATA_CLASS(LayerNormV4_422, LayerNormV4TilingDataWelford) |
| 133 | 133 | ||
| 134 | +BEGIN_TILING_DATA_DEF(LayerNormV4TilingDataSplitR) | ||
| 135 | +TILING_DATA_FIELD_DEF(int64_t, M); // 输入tensor的行 | ||
| 136 | +TILING_DATA_FIELD_DEF(int64_t, N); // 输入tensor的列,即reduce的轴 | ||
| 137 | +TILING_DATA_FIELD_DEF(int64_t, rAlign); // r对齐的大小 | ||
| 138 | +TILING_DATA_FIELD_DEF(int64_t, numBlocks); // 实际使用的core数量 | ||
| 139 | +TILING_DATA_FIELD_DEF(int64_t, mainBlockCount); // 整核的数量 | ||
| 140 | +TILING_DATA_FIELD_DEF(int64_t, mainBlockFactor); // 整核处理的row大小 | ||
| 141 | +TILING_DATA_FIELD_DEF(int64_t, tailBlockFactor); // 尾核处理的row大小 | ||
| 142 | +TILING_DATA_FIELD_DEF(int64_t, tileLength); // tile块的元素个数 | ||
| 143 | +TILING_DATA_FIELD_DEF(int64_t, welfordTempSize); // welford临时buffer的大小 | ||
| 144 | +TILING_DATA_FIELD_DEF(int64_t, welfordUpdateTimes); // welford update的次数 | ||
| 145 | +TILING_DATA_FIELD_DEF(int64_t, welfordUpdateTail); // welford update的尾数 | ||
| 146 | +TILING_DATA_FIELD_DEF(int64_t, nullptrGamma); | ||
| 147 | +TILING_DATA_FIELD_DEF(int64_t, nullptrBeta); | ||
| 148 | +TILING_DATA_FIELD_DEF(int64_t, apiTempBufferSize); | ||
| 149 | +TILING_DATA_FIELD_DEF(float, epsilon); | ||
| 150 | +TILING_DATA_FIELD_DEF(int64_t, splitR); | ||
| 151 | +END_TILING_DATA_DEF; | ||
| 152 | + | ||
| 153 | +REGISTER_TILING_DATA_CLASS(LayerNormV4_800, LayerNormV4TilingDataSplitR) | ||
| 154 | +REGISTER_TILING_DATA_CLASS(LayerNormV4_810, LayerNormV4TilingDataSplitR) | ||
| 155 | +REGISTER_TILING_DATA_CLASS(LayerNormV4_811, LayerNormV4TilingDataSplitR) | ||
| 156 | +REGISTER_TILING_DATA_CLASS(LayerNormV4_820, LayerNormV4TilingDataSplitR) | ||
| 157 | +REGISTER_TILING_DATA_CLASS(LayerNormV4_822, LayerNormV4TilingDataSplitR) | ||
| 158 | + | ||
| 134 | BEGIN_TILING_DATA_DEF(LayerNormV4TilingDataRegBaseTwoPassPerf) | 159 | BEGIN_TILING_DATA_DEF(LayerNormV4TilingDataRegBaseTwoPassPerf) |
| 135 | TILING_DATA_FIELD_DEF(int64_t, a); | 160 | TILING_DATA_FIELD_DEF(int64_t, a); |
| 136 | TILING_DATA_FIELD_DEF(int64_t, aBlockFactor); | 161 | TILING_DATA_FIELD_DEF(int64_t, aBlockFactor); |
| @@ -0,0 +1,266 @@ | |||
| 1 | +/** | ||
| 2 | + * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | + * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | + * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | + * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | + * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | + * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | + */ | ||
| 10 | + | ||
| 11 | +/*! | ||
| 12 | + * \file layer_norm_v4_split_r.h | ||
| 13 | + * \brief | ||
| 14 | + */ | ||
| 15 | +#ifndef LAYER_NORM_V4_SPLIT_R_H | ||
| 16 | +#define LAYER_NORM_V4_SPLIT_R_H | ||
| 17 | + | ||
| 18 | +#include "layer_norm_v4_welford_base.h" | ||
| 19 | +#include "../../norm_common/reduce_common_regbase.h" | ||
| 20 | + | ||
| 21 | +namespace LayerNormV4 { | ||
| 22 | +// Build weighted contributions before invoking the common length-aware reduction. | ||
| 23 | +class LayerNormV4SplitRCommon { | ||
| 24 | +public: | ||
| 25 | + __aicore__ inline static void FillCountBlock(__ubuf__ float* dst, AscendC::Reg::RegTensor<float>& value, | ||
| 26 | + AscendC::Reg::MaskReg& full, AscendC::Reg::MaskReg& active, | ||
| 27 | + float count, uint32_t num, uint16_t loops, uint32_t vl) | ||
| 28 | + { | ||
| 29 | + AscendC::Reg::Duplicate(value, count, full); | ||
| 30 | + for (uint16_t i = 0; i < loops; ++i) { | ||
| 31 | + active = AscendC::Reg::UpdateMask<float>(num); | ||
| 32 | + AscendC::Reg::StoreAlign(dst + i * vl, value, active); | ||
| 33 | + } | ||
| 34 | + } | ||
| 35 | + | ||
| 36 | + __aicore__ inline static void Merge(LocalTensor<float>& meanOut, LocalTensor<float>& varianceOut, | ||
| 37 | + LocalTensor<float>& means, LocalTensor<float>& variances, | ||
| 38 | + LocalTensor<float>& counts, LocalTensor<float>& contributions, | ||
| 39 | + LocalTensor<float>& reduceTmp, uint32_t splits, float inverseCount) | ||
| 40 | + { | ||
| 41 | + auto* meanAddr = (__ubuf__ float*)means.GetPhyAddr(); | ||
| 42 | + auto* varAddr = (__ubuf__ float*)variances.GetPhyAddr(); | ||
| 43 | + auto* countAddr = (__ubuf__ float*)counts.GetPhyAddr(); | ||
| 44 | + auto* contributionAddr = (__ubuf__ float*)contributions.GetPhyAddr(); | ||
| 45 | + auto* meanOutAddr = (__ubuf__ float*)meanOut.GetPhyAddr(); | ||
| 46 | + constexpr uint32_t vl = NormCommon::V_LENGTH; | ||
| 47 | + uint32_t foldPoint = vl; | ||
| 48 | + while (foldPoint <= splits / 2) { | ||
| 49 | + foldPoint *= 2; | ||
| 50 | + } | ||
| 51 | + // RegBase vector loops require a uint16_t induction variable and a | ||
| 52 | + // bound divisible by the step; the mask covers the aligned tail. | ||
| 53 | + uint32_t splitsAligned = (splits + vl - 1) / vl * vl; | ||
| 54 | + __VEC_SCOPE__ | ||
| 55 | + { | ||
| 56 | + using namespace AscendC::Reg; | ||
| 57 | + RegTensor<float> mean, weight, reference, delta; | ||
| 58 | + LoadAlign<float, LoadDist::DIST_BRC_B32>(reference, meanAddr); | ||
| 59 | + uint32_t remaining = splits; | ||
| 60 | + for (uint16_t offset = 0; offset < splitsAligned; offset += vl) { | ||
| 61 | + MaskReg active = UpdateMask<float>(remaining); | ||
| 62 | + LoadAlign(mean, meanAddr + offset); | ||
| 63 | + LoadAlign(weight, countAddr + offset); | ||
| 64 | + Muls(weight, weight, inverseCount, active); | ||
| 65 | + // Retain the centered-mean fix for large-offset inputs. | ||
| 66 | + Sub(delta, mean, reference, active); | ||
| 67 | + Mul(delta, delta, weight, active); | ||
| 68 | + StoreAlign(contributionAddr + offset, delta, active); | ||
| 69 | + } | ||
| 70 | + LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>(); | ||
| 71 | + } | ||
| 72 | + NormCommon::NormCommonRegbase::CalculateReduceSum(contributions, meanOut, reduceTmp, splits, foldPoint); | ||
| 73 | + __VEC_SCOPE__ | ||
| 74 | + { | ||
| 75 | + using namespace AscendC::Reg; | ||
| 76 | + RegTensor<float> mean, variance, weight, reference, delta, mergedMean; | ||
| 77 | + MaskReg first = CreateMask<float, MaskPattern::VL1>(); | ||
| 78 | + LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>(); | ||
| 79 | + LoadAlign<float, LoadDist::DIST_BRC_B32>(reference, meanAddr); | ||
| 80 | + LoadAlign<float, LoadDist::DIST_BRC_B32>(mergedMean, meanOutAddr); | ||
| 81 | + Add(mergedMean, mergedMean, reference, first); | ||
| 82 | + StoreAlign(meanOutAddr, mergedMean, first); | ||
| 83 | + LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>(); | ||
| 84 | + LoadAlign<float, LoadDist::DIST_BRC_B32>(mergedMean, meanOutAddr); | ||
| 85 | + uint32_t remaining = splits; | ||
| 86 | + for (uint16_t offset = 0; offset < splitsAligned; offset += vl) { | ||
| 87 | + MaskReg active = UpdateMask<float>(remaining); | ||
| 88 | + LoadAlign(mean, meanAddr + offset); | ||
| 89 | + LoadAlign(variance, varAddr + offset); | ||
| 90 | + LoadAlign(weight, countAddr + offset); | ||
| 91 | + // Weight before multiplying variance to avoid count-amplified overflow. | ||
| 92 | + Muls(weight, weight, inverseCount, active); | ||
| 93 | + Sub(delta, mean, mergedMean, active); | ||
| 94 | + Mul(delta, delta, delta, active); | ||
| 95 | + Mul(delta, delta, weight, active); | ||
| 96 | + Mul(variance, variance, weight, active); | ||
| 97 | + Add(variance, variance, delta, active); | ||
| 98 | + StoreAlign(contributionAddr + offset, variance, active); | ||
| 99 | + } | ||
| 100 | + LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>(); | ||
| 101 | + } | ||
| 102 | + NormCommon::NormCommonRegbase::CalculateReduceSum(contributions, varianceOut, reduceTmp, splits, foldPoint); | ||
| 103 | + } | ||
| 104 | +}; | ||
| 105 | + | ||
| 106 | +template <typename T, typename U, typename M> | ||
| 107 | +class LayerNormV4RegbaseSplitR : public LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataSplitR> { | ||
| 108 | + using Base = LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataSplitR>; | ||
| 109 | + | ||
| 110 | +public: | ||
| 111 | + __aicore__ inline LayerNormV4RegbaseSplitR(const LayerNormV4TilingDataSplitR* tilingData, TPipe* pipeIn) | ||
| 112 | + : Base(tilingData, pipeIn) | ||
| 113 | + {} | ||
| 114 | + | ||
| 115 | + __aicore__ inline void Init(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd, | ||
| 116 | + GM_ADDR workspace) | ||
| 117 | + { | ||
| 118 | + row_ = GetBlockIdx() / this->td_->splitR; | ||
| 119 | + split_ = GetBlockIdx() % this->td_->splitR; | ||
| 120 | + segmentStart_ = SegmentStart(split_); | ||
| 121 | + segmentCount_ = SegmentCount(split_); | ||
| 122 | + this->xGm.SetGlobalBuffer((__gm__ T*)x + row_ * this->td_->N + segmentStart_, segmentCount_); | ||
| 123 | + this->yGm.SetGlobalBuffer((__gm__ T*)y + row_ * this->td_->N + segmentStart_, segmentCount_); | ||
| 124 | + this->meanGm.SetGlobalBuffer((__gm__ M*)mean + row_, 1); | ||
| 125 | + this->rstdGm.SetGlobalBuffer((__gm__ M*)rstd + row_, 1); | ||
| 126 | + partialGm_.SetGlobalBuffer((__gm__ float*)workspace); | ||
| 127 | + // Each GM record occupies a UB block; compact copies pack lane 0 from | ||
| 128 | + // every record into vector-aligned mean/variance planes. | ||
| 129 | + partialHalf_ = (this->td_->splitR + VECTOR_FLOATS - 1) / VECTOR_FLOATS * VECTOR_FLOATS; | ||
| 130 | + this->pipe_->InitBuffer(partialIn_, 1, 2 * partialHalf_ * sizeof(float)); | ||
| 131 | + this->InitCommon(gamma, beta); | ||
| 132 | + } | ||
| 133 | + | ||
| 134 | + __aicore__ inline void Process() | ||
| 135 | + { | ||
| 136 | + this->ResetCache(); | ||
| 137 | + this->mean_ = this->welfordTempBuffer.template Get<float>(); | ||
| 138 | + this->variance_ = this->mean_[this->td_->tileLength]; | ||
| 139 | + this->varianceTensor = this->variance_[this->td_->tileLength]; | ||
| 140 | + this->shared_ = this->apiTempBuffer.template Get<uint8_t>(); | ||
| 141 | + this->WelfordInitialize(this->mean_, this->variance_, this->td_->tileLength); | ||
| 142 | + int64_t loops = segmentCount_ / this->td_->tileLength; | ||
| 143 | + int64_t tail = segmentCount_ % this->td_->tileLength; | ||
| 144 | + for (int64_t i = 0; i < loops; ++i) { | ||
| 145 | + this->ProcessWelfordUpdate(i * this->td_->tileLength, this->td_->tileLength); | ||
| 146 | + } | ||
| 147 | + if (tail > 0) { | ||
| 148 | + this->ProcessWelfordUpdate(loops * this->td_->tileLength, tail); | ||
| 149 | + } | ||
| 150 | + WelfordFinalizePara para; | ||
| 151 | + para.rnLength = this->welfordCount; | ||
| 152 | + para.abLength = this->td_->tileLength; | ||
| 153 | + para.headCount = this->welfordCount; | ||
| 154 | + para.headCountLength = tail; | ||
| 155 | + para.tailCount = this->welfordCount - (tail > 0); | ||
| 156 | + para.tailCountLength = this->td_->tileLength - tail; | ||
| 157 | + para.abRec = 1.0f / static_cast<float>(this->td_->tileLength); | ||
| 158 | + para.rRec = 1.0f / static_cast<float>(segmentCount_); | ||
| 159 | + WelfordFinalize<true>(this->meanTensor, this->varianceTensor, this->mean_, this->variance_, this->shared_, | ||
| 160 | + para); | ||
| 161 | + | ||
| 162 | + // Publish mean and population variance in separate GM planes. | ||
| 163 | + // SyncAll includes the preceding MTE3 writes before peer reads. | ||
| 164 | + auto vToMte3 = this->pipe_->FetchEventID(HardEvent::V_MTE3); | ||
| 165 | + SetFlag<HardEvent::V_MTE3>(vToMte3); | ||
| 166 | + WaitFlag<HardEvent::V_MTE3>(vToMte3); | ||
| 167 | + int64_t planeLength = this->td_->M * this->td_->splitR * PARTIAL_FLOATS; | ||
| 168 | + DataCopyExtParams partialCopyParams{1, PARTIAL_FLOATS * sizeof(float), 0, 0, 0}; | ||
| 169 | + DataCopyPad(partialGm_[GetBlockIdx() * PARTIAL_FLOATS], this->meanTensor, partialCopyParams); | ||
| 170 | + DataCopyPad(partialGm_[planeLength + GetBlockIdx() * PARTIAL_FLOATS], this->varianceTensor, partialCopyParams); | ||
| 171 | + SyncAll(); | ||
[P1][ascendc-red-line][RED-7] SplitR 需要启用硬同步所需的 Kernel 类型(报告严重级别:高) 这里新增了无参数硬件 DAV_3510 上 建议:为这五个 SplitR key 配置 依据:官方核间同步类型约束。此结论为源码、SDK 和文档契约核实,未声称已在 NPU 实测卡死。 检视版本: ![]() ![]() | |||
| 172 | + | ||
| 173 | + auto all = partialIn_.template AllocTensor<float>(); | ||
| 174 | + auto allMean = all; | ||
| 175 | + auto allVar = all[partialHalf_]; | ||
| 176 | + int64_t rowOffset = row_ * this->td_->splitR * PARTIAL_FLOATS; | ||
| 177 | + DataCopyExtParams compactCopy{static_cast<uint16_t>(this->td_->splitR), sizeof(float), | ||
| 178 | + static_cast<uint32_t>((PARTIAL_FLOATS - 1) * sizeof(float)), 0, 0}; | ||
| 179 | + DataCopyPadExtParams<float> noPadding{false, 0, 0, 0}; | ||
| 180 | + DataCopyPad<float, PaddingMode::Compact>(allMean, partialGm_[rowOffset], compactCopy, noPadding); | ||
| 181 | + DataCopyPad<float, PaddingMode::Compact>(allVar, partialGm_[planeLength + rowOffset], compactCopy, noPadding); | ||
| 182 | + partialIn_.EnQue(all); | ||
| 183 | + all = partialIn_.template DeQue<float>(); | ||
| 184 | + allMean = all; | ||
| 185 | + allVar = all[partialHalf_]; | ||
| 186 | + // The local Welford state is dead; reuse mean_ for compact segment counts. | ||
| 187 | + FillMergeCounts(); | ||
这里直接紧凑型搬运过来,使用DataCopyPad的compact模式,例如S1S2,不要S1..(28b)...S2 ![]() ![]() | |||
| 188 | + // With splitR <= 16, the dead tile-sized M2 and 256-float rstd buffers | ||
| 189 | + // cover the contributions and common reduction scratch, respectively. | ||
| 190 | + LayerNormV4SplitRCommon::Merge(this->meanTensor, this->varianceTensor, allMean, allVar, this->mean_, | ||
| 191 | + this->variance_, this->rstdTensor, this->td_->splitR, | ||
| 192 | + 1.0f / static_cast<float>(this->td_->N)); | ||
| 193 | + partialIn_.FreeTensor(all); | ||
| 194 | + for (int64_t offset = 0; offset < segmentCount_; offset += this->td_->tileLength) { | ||
| 195 | + int64_t length = segmentCount_ - offset; | ||
| 196 | + length = length < this->td_->tileLength ? length : this->td_->tileLength; | ||
| 197 | + this->ProcessNormalize(offset, segmentStart_ + offset, length); | ||
| 198 | + } | ||
| 199 | + if (split_ == 0) { | ||
| 200 | + this->cacheCount = 1; | ||
| 201 | + this->paramAddr = 0; | ||
| 202 | + this->RefreshCache(); | ||
| 203 | + } else { | ||
| 204 | + this->outQueueMean.FreeTensor(this->meanTensor); | ||
| 205 | + this->outQueueRstd.FreeTensor(this->rstdTensor); | ||
| 206 | + } | ||
| 207 | + } | ||
| 208 | + | ||
| 209 | +private: | ||
| 210 | + __aicore__ inline void FillMergeCounts() | ||
| 211 | + { | ||
| 212 | + int64_t units = (this->td_->N + SPLIT_ALIGNMENT - 1) / SPLIT_ALIGNMENT; | ||
| 213 | + float baseCount = static_cast<float>(units / this->td_->splitR * SPLIT_ALIGNMENT); | ||
| 214 | + float lastCount = static_cast<float>(SegmentCount(this->td_->splitR - 1)); | ||
| 215 | + uint32_t longCount = units % this->td_->splitR; | ||
| 216 | + uint32_t splitCount = this->td_->splitR; | ||
| 217 | + uint16_t firstLoopCount = (splitCount + VECTOR_FLOATS - 1) / VECTOR_FLOATS; | ||
| 218 | + uint16_t secondLoopCount = (splitCount - 1 + VECTOR_FLOATS - 1) / VECTOR_FLOATS; | ||
| 219 | + uint16_t thirdLoopCount = (longCount + VECTOR_FLOATS - 1) / VECTOR_FLOATS; | ||
| 220 | + auto* counts = (__ubuf__ float*)this->mean_.GetPhyAddr(); | ||
| 221 | + // Overwrite order: last segment, ordinary segments, then the longer | ||
| 222 | + // leading segments. | ||
| 223 | + __VEC_SCOPE__ | ||
| 224 | + { | ||
| 225 | + AscendC::Reg::RegTensor<float> value; | ||
| 226 | + auto mask = AscendC::Reg::CreateMask<float, AscendC::Reg::MaskPattern::ALL>(); | ||
| 227 | + AscendC::Reg::MaskReg active; | ||
| 228 | + LayerNormV4SplitRCommon::FillCountBlock(counts, value, mask, active, lastCount, splitCount, firstLoopCount, | ||
| 229 | + VECTOR_FLOATS); | ||
| 230 | + LayerNormV4SplitRCommon::FillCountBlock(counts, value, mask, active, baseCount, splitCount - 1, | ||
| 231 | + secondLoopCount, VECTOR_FLOATS); | ||
| 232 | + LayerNormV4SplitRCommon::FillCountBlock(counts, value, mask, active, baseCount + SPLIT_ALIGNMENT, longCount, | ||
| 233 | + thirdLoopCount, VECTOR_FLOATS); | ||
| 234 | + } | ||
| 235 | + } | ||
| 236 | + | ||
| 237 | + __aicore__ inline int64_t SegmentStart(int64_t split) const | ||
| 238 | + { | ||
| 239 | + int64_t units = (this->td_->N + SPLIT_ALIGNMENT - 1) / SPLIT_ALIGNMENT; | ||
| 240 | + int64_t quotient = units / this->td_->splitR; | ||
| 241 | + int64_t remainder = units % this->td_->splitR; | ||
| 242 | + return (split * quotient + (split < remainder ? split : remainder)) * SPLIT_ALIGNMENT; | ||
| 243 | + } | ||
| 244 | + | ||
| 245 | + __aicore__ inline int64_t SegmentCount(int64_t split) const | ||
| 246 | + { | ||
| 247 | + int64_t end = SegmentStart(split + 1); | ||
| 248 | + end = end < this->td_->N ? end : this->td_->N; | ||
| 249 | + return end - SegmentStart(split); | ||
| 250 | + } | ||
| 251 | + | ||
| 252 | + static constexpr int64_t SPLIT_ALIGNMENT = 16; | ||
| 253 | + static constexpr int64_t PARTIAL_FLOATS = platform::GetUbBlockSize() / sizeof(float); | ||
| 254 | + static constexpr int64_t VECTOR_FLOATS = LayerNormV4Regbase::GetVRegSize() / sizeof(float); | ||
| 255 | + int64_t partialHalf_ = 0; | ||
| 256 | + int64_t row_ = 0; | ||
| 257 | + int64_t split_ = 0; | ||
| 258 | + int64_t segmentStart_ = 0; | ||
| 259 | + int64_t segmentCount_ = 0; | ||
| 260 | + GlobalTensor<float> partialGm_; | ||
| 261 | + TQue<QuePosition::VECIN, 1> partialIn_; | ||
| 262 | +}; | ||
| 263 | + | ||
| 264 | +} // namespace LayerNormV4 | ||
| 265 | + | ||
| 266 | +#endif // LAYER_NORM_V4_SPLIT_R_H | ||
| @@ -12,385 +12,108 @@ | |||
| 12 | * \file layer_norm_v4_welford.h | 12 | * \file layer_norm_v4_welford.h |
| 13 | * \brief | 13 | * \brief |
| 14 | */ | 14 | */ |
| 15 | - | ||
| 16 | #ifndef LAYER_NORM_V4_WELFORD_H | 15 | #ifndef LAYER_NORM_V4_WELFORD_H |
| 17 | #define LAYER_NORM_V4_WELFORD_H | 16 | #define LAYER_NORM_V4_WELFORD_H |
| 18 | 17 | ||
| 19 | -#include "layer_norm_v4_regbase_common.h" | 18 | +#include "layer_norm_v4_welford_base.h" |
| 20 | -#include "kernel_tiling/kernel_tiling.h" | ||
| 21 | -#include "kernel_operator.h" | ||
| 22 | -#include "../../inc/platform.h" | ||
| 23 | 19 | ||
| 24 | namespace LayerNormV4 { | 20 | namespace LayerNormV4 { |
| 25 | -using namespace AscendC; | ||
| 26 | -using AscendC::Reg::StoreAlign; | ||
| 27 | - | ||
| 28 | -namespace LayerNormV4Regbase { | ||
| 29 | -__aicore__ inline constexpr uint32_t GetVRegSize() | ||
| 30 | -{ | ||
| 31 | -#if __CCE_AICORE__ == 310 || __CCE_AICORE__ == 920 || __NPU_ARCH == 5102 | ||
| 32 | - return AscendC::VECTOR_REG_WIDTH; | ||
| 33 | -#else | ||
| 34 | - return 256U; | ||
| 35 | -#endif | ||
| 36 | -} | ||
| 37 | -} // namespace LayerNormV4Regbase | ||
| 38 | - | ||
| 39 | -template <typename T> | ||
| 40 | -__aicore__ inline void CopyIn(const LocalTensor<T>& dstTensor, const GlobalTensor<T>& srcTensor, const int64_t rowSize) | ||
| 41 | -{ | ||
| 42 | - // CopyIn | ||
| 43 | - DataCopyExtParams params; | ||
| 44 | - params.blockCount = 1; | ||
| 45 | - params.blockLen = rowSize * sizeof(T); | ||
| 46 | - DataCopyPadExtParams<T> padParams; | ||
| 47 | - padParams.isPad = false; | ||
| 48 | - DataCopyPad(dstTensor, srcTensor, params, padParams); | ||
| 49 | -} | ||
| 50 | - | ||
| 51 | -template <typename T> | ||
| 52 | -__aicore__ inline void CopyOut(const GlobalTensor<T>& dstTensor, const LocalTensor<T>& srcTensor, const int64_t rowSize) | ||
| 53 | -{ | ||
| 54 | - // CopyOut | ||
| 55 | - DataCopyExtParams params; | ||
| 56 | - params.blockCount = 1; | ||
| 57 | - params.blockLen = rowSize * sizeof(T); | ||
| 58 | - DataCopyPad(dstTensor, srcTensor, params); | ||
| 59 | -} | ||
| 60 | - | ||
| 61 | template <typename T, typename U, typename M> | 21 | template <typename T, typename U, typename M> |
| 62 | -class LayerNormV4RegbaseWelford { | 22 | +class LayerNormV4RegbaseWelford : public LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataWelford> { |
| 23 | + using Base = LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataWelford>; | ||
| 24 | + | ||
| 63 | public: | 25 | public: |
| 64 | __aicore__ inline LayerNormV4RegbaseWelford(const LayerNormV4TilingDataWelford* tilingData, TPipe* pipeIn) | 26 | __aicore__ inline LayerNormV4RegbaseWelford(const LayerNormV4TilingDataWelford* tilingData, TPipe* pipeIn) |
| 65 | - { | 27 | + : Base(tilingData, pipeIn) |
| 66 | - // Constructor | 28 | + {} |
| 67 | - pipe_ = pipeIn; | ||
| 68 | - td_ = tilingData; | ||
| 69 | - } | ||
| 70 | 29 | ||
| 71 | __aicore__ inline void Init(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd) | 30 | __aicore__ inline void Init(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd) |
| 72 | { | 31 | { |
| 73 | - // Init | 32 | + if (GetBlockIdx() >= this->td_->numBlocks) { |
| 74 | - if (GetBlockIdx() >= td_->numBlocks) { | ||
| 75 | return; | 33 | return; |
| 76 | } | 34 | } |
| 77 | - | 35 | + currentBlockFactor = (GetBlockIdx() == this->td_->mainBlockCount) ? this->td_->tailBlockFactor : |
| 78 | - // init global memory | 36 | + this->td_->mainBlockFactor; |
| 79 | - currentBlockFactor = (GetBlockIdx() == td_->mainBlockCount) ? td_->tailBlockFactor : td_->mainBlockFactor; | 37 | + int64_t fmOffset = GetBlockIdx() * this->td_->mainBlockFactor * this->td_->N; |
| 80 | - int64_t fmOffset = GetBlockIdx() * td_->mainBlockFactor * td_->N; | 38 | + int64_t paramOffset = GetBlockIdx() * this->td_->mainBlockFactor; |
| 81 | - int64_t paramOffset = GetBlockIdx() * td_->mainBlockFactor; | 39 | + int64_t fmSize = currentBlockFactor * this->td_->N; |
| 82 | - int64_t fmSize = currentBlockFactor * td_->N; | ||
| 83 | int64_t paramSize = currentBlockFactor; | 40 | int64_t paramSize = currentBlockFactor; |
| 84 | 41 | ||
| 85 | - xGm.SetGlobalBuffer((__gm__ T*)x + fmOffset, fmSize); | 42 | + this->xGm.SetGlobalBuffer((__gm__ T*)x + fmOffset, fmSize); |
| 86 | - yGm.SetGlobalBuffer((__gm__ T*)y + fmOffset, fmSize); | 43 | + this->yGm.SetGlobalBuffer((__gm__ T*)y + fmOffset, fmSize); |
| 87 | - meanGm.SetGlobalBuffer((__gm__ M*)mean + paramOffset, paramSize); | 44 | + this->meanGm.SetGlobalBuffer((__gm__ M*)mean + paramOffset, paramSize); |
| 88 | - rstdGm.SetGlobalBuffer((__gm__ M*)rstd + paramOffset, paramSize); | 45 | + this->rstdGm.SetGlobalBuffer((__gm__ M*)rstd + paramOffset, paramSize); |
| 89 | - if (td_->nullptrGamma == 0) { | 46 | + this->InitCommon(gamma, beta); |
| 90 | - gammaGm.SetGlobalBuffer((__gm__ U*)gamma, td_->N); | ||
| 91 | - } | ||
| 92 | - if (td_->nullptrBeta == 0) { | ||
| 93 | - betaGm.SetGlobalBuffer((__gm__ U*)beta, td_->N); | ||
| 94 | - } | ||
| 95 | - | ||
| 96 | - // init local memory | ||
| 97 | - pipe_->InitBuffer(inQueueX, DOUBLE_BUFFER, td_->tileLength * sizeof(T)); | ||
| 98 | - pipe_->InitBuffer(outQueueY, DOUBLE_BUFFER, td_->tileLength * sizeof(T)); | ||
| 99 | - pipe_->InitBuffer(outQueueMean, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float)); | ||
| 100 | - pipe_->InitBuffer(outQueueRstd, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float)); | ||
| 101 | - pipe_->InitBuffer(welfordTempBuffer, td_->welfordTempSize); | ||
| 102 | - if (td_->nullptrGamma == 0) { | ||
| 103 | - pipe_->InitBuffer(inQueueGamma, DOUBLE_BUFFER, td_->tileLength * sizeof(U)); | ||
| 104 | - } | ||
| 105 | - if (td_->nullptrBeta == 0) { | ||
| 106 | - pipe_->InitBuffer(inQueueBeta, DOUBLE_BUFFER, td_->tileLength * sizeof(U)); | ||
| 107 | - } | ||
| 108 | - pipe_->InitBuffer(apiTempBuffer, td_->apiTempBufferSize); | ||
| 109 | } | 47 | } |
| 110 | 48 | ||
| 111 | __aicore__ inline void Process() | 49 | __aicore__ inline void Process() |
| 112 | { | 50 | { |
| 113 | // Process | 51 | // Process |
| 114 | - if (GetBlockIdx() >= td_->numBlocks) { | 52 | + if (GetBlockIdx() >= this->td_->numBlocks) { |
| 115 | return; | 53 | return; |
| 116 | } | 54 | } |
| 117 | 55 | ||
| 118 | - ResetCache(); | 56 | + this->ResetCache(); |
| 119 | - paramAddr = 0; | 57 | + this->paramAddr = 0; |
| 120 | - mean_ = welfordTempBuffer.Get<float>(); | 58 | + this->mean_ = this->welfordTempBuffer.template Get<float>(); |
| 121 | - variance_ = mean_[td_->tileLength]; | 59 | + this->variance_ = this->mean_[this->td_->tileLength]; |
| 122 | - varianceTensor = variance_[td_->tileLength]; | 60 | + this->varianceTensor = this->variance_[this->td_->tileLength]; |
| 123 | - shared_ = apiTempBuffer.Get<uint8_t>(); | 61 | + this->shared_ = this->apiTempBuffer.template Get<uint8_t>(); |
| 124 | 62 | ||
| 125 | for (int64_t i = 0; i < currentBlockFactor; ++i) { | 63 | for (int64_t i = 0; i < currentBlockFactor; ++i) { |
| 126 | // Welford Algorithm | 64 | // Welford Algorithm |
| 127 | - WelfordInitialize(mean_, variance_, td_->tileLength); | 65 | + this->WelfordInitialize(this->mean_, this->variance_, this->td_->tileLength); |
| 128 | - for (int64_t welfordUpdateCount = 0; welfordUpdateCount < td_->welfordUpdateTimes; welfordUpdateCount++) { | 66 | + for (int64_t welfordUpdateCount = 0; welfordUpdateCount < this->td_->welfordUpdateTimes; |
| 129 | - int64_t offset = i * td_->N + welfordUpdateCount * td_->tileLength; | 67 | + welfordUpdateCount++) { |
| 130 | - ProcessWelfordUpdate(offset, td_->tileLength); | 68 | + int64_t offset = i * this->td_->N + welfordUpdateCount * this->td_->tileLength; |
| 69 | + this->ProcessWelfordUpdate(offset, this->td_->tileLength); | ||
| 131 | } | 70 | } |
| 132 | - if (td_->welfordUpdateTail > 0) { | 71 | + if (this->td_->welfordUpdateTail > 0) { |
| 133 | - int64_t offset = i * td_->N + td_->welfordUpdateTimes * td_->tileLength; | 72 | + int64_t offset = i * this->td_->N + this->td_->welfordUpdateTimes * this->td_->tileLength; |
| 134 | - ProcessWelfordUpdate(offset, td_->welfordUpdateTail); | 73 | + this->ProcessWelfordUpdate(offset, this->td_->welfordUpdateTail); |
| 135 | } | 74 | } |
| 136 | 75 | ||
| 137 | WelfordFinalizePara para; | 76 | WelfordFinalizePara para; |
| 138 | - para.rnLength = welfordCount; | 77 | + para.rnLength = this->welfordCount; |
| 139 | - para.abLength = td_->tileLength; | 78 | + para.abLength = this->td_->tileLength; |
| 140 | - para.headCount = welfordCount; | 79 | + para.headCount = this->welfordCount; |
| 141 | - para.headCountLength = td_->welfordUpdateTail; | 80 | + para.headCountLength = this->td_->welfordUpdateTail; |
| 142 | - para.tailCount = welfordCount - (td_->welfordUpdateTail > 0); | 81 | + para.tailCount = this->welfordCount - (this->td_->welfordUpdateTail > 0); |
| 143 | - para.tailCountLength = td_->tileLength - td_->welfordUpdateTail; | 82 | + para.tailCountLength = this->td_->tileLength - this->td_->welfordUpdateTail; |
| 144 | - para.abRec = 1.0f / static_cast<float>(td_->tileLength); | 83 | + para.abRec = 1.0f / static_cast<float>(this->td_->tileLength); |
| 145 | - para.rRec = 1.0f / static_cast<float>(td_->N); | 84 | + para.rRec = 1.0f / static_cast<float>(this->td_->N); |
| 146 | - WelfordFinalize<true>(meanTensor[cacheCount], varianceTensor[cacheCount], mean_, variance_, shared_, para); | 85 | + WelfordFinalize<true>(this->meanTensor[this->cacheCount], this->varianceTensor[this->cacheCount], |
| 86 | + this->mean_, this->variance_, this->shared_, para); | ||
| 147 | 87 | ||
| 148 | // Normalize | 88 | // Normalize |
| 149 | - for (int64_t welfordUpdateCount = 0; welfordUpdateCount < td_->welfordUpdateTimes; welfordUpdateCount++) { | 89 | + for (int64_t welfordUpdateCount = 0; welfordUpdateCount < this->td_->welfordUpdateTimes; |
| 150 | - int64_t fmOffset = i * td_->N + welfordUpdateCount * td_->tileLength; | 90 | + welfordUpdateCount++) { |
| 151 | - int64_t paramOffset = welfordUpdateCount * td_->tileLength; | 91 | + int64_t fmOffset = i * this->td_->N + welfordUpdateCount * this->td_->tileLength; |
| 152 | - ProcessNormalize(fmOffset, paramOffset, td_->tileLength); | 92 | + int64_t paramOffset = welfordUpdateCount * this->td_->tileLength; |
| 93 | + this->ProcessNormalize(fmOffset, paramOffset, this->td_->tileLength); | ||
| 153 | } | 94 | } |
| 154 | - if (td_->welfordUpdateTail > 0) { | 95 | + if (this->td_->welfordUpdateTail > 0) { |
| 155 | - int64_t fmOffset = i * td_->N + td_->welfordUpdateTimes * td_->tileLength; | 96 | + int64_t fmOffset = i * this->td_->N + this->td_->welfordUpdateTimes * this->td_->tileLength; |
| 156 | - int64_t paramOffset = td_->welfordUpdateTimes * td_->tileLength; | 97 | + int64_t paramOffset = this->td_->welfordUpdateTimes * this->td_->tileLength; |
| 157 | - ProcessNormalize(fmOffset, paramOffset, td_->welfordUpdateTail); | 98 | + this->ProcessNormalize(fmOffset, paramOffset, this->td_->welfordUpdateTail); |
| 158 | } | 99 | } |
| 159 | 100 | ||
| 160 | - cacheCount++; | 101 | + this->cacheCount++; |
| 161 | // check cache buffer | 102 | // check cache buffer |
| 162 | - if (cacheCount >= AGGREGATION_COUNT) { | 103 | + if (this->cacheCount >= this->AGGREGATION_COUNT) { |
| 163 | - RefreshCache(); | 104 | + this->RefreshCache(); |
| 164 | - ResetCache(); | 105 | + this->ResetCache(); |
| 165 | } | 106 | } |
| 166 | } | 107 | } |
| 167 | 108 | ||
| 168 | // refresh cache | 109 | // refresh cache |
| 169 | - if (cacheCount > 0) { | 110 | + if (this->cacheCount > 0) { |
| 170 | - RefreshCache(); | 111 | + this->RefreshCache(); |
| 171 | } | 112 | } |
| 172 | } | 113 | } |
| 173 | 114 | ||
| 174 | private: | 115 | private: |
| 175 | - __aicore__ inline void ResetCache() | ||
| 176 | - { | ||
| 177 | - // ResetCache | ||
| 178 | - cacheCount = 0; | ||
| 179 | - meanTensor = outQueueMean.template AllocTensor<float>(); | ||
| 180 | - rstdTensor = outQueueRstd.template AllocTensor<float>(); | ||
| 181 | - } | ||
| 182 | - | ||
| 183 | - __aicore__ inline void RefreshCache() | ||
| 184 | - { | ||
| 185 | - if constexpr (!IsSameType<M, float>::value) { | ||
| 186 | - // float to bfloat16 or float16, input continue and output each repeat have only half value | ||
| 187 | - CastBatchMeanRstdToDtype<M>((__ubuf__ float*)meanTensor.GetPhyAddr(), | ||
| 188 | - (__ubuf__ float*)rstdTensor.GetPhyAddr(), (__ubuf__ M*)meanTensor.GetPhyAddr(), | ||
| 189 | - (__ubuf__ M*)rstdTensor.GetPhyAddr(), cacheCount); | ||
| 190 | - outQueueMean.EnQue(meanTensor); | ||
| 191 | - outQueueRstd.EnQue(rstdTensor); | ||
| 192 | - meanTensor = outQueueMean.template DeQue<float>(); | ||
| 193 | - rstdTensor = outQueueRstd.template DeQue<float>(); | ||
| 194 | - | ||
| 195 | - // VL_F32 | ||
| 196 | - uint32_t castDmaCount = static_cast<uint32_t>(cacheCount); | ||
| 197 | - uint32_t castDmaLoops = static_cast<uint32_t>(castDmaCount / VL_F32); | ||
| 198 | - if (castDmaLoops > 0) { | ||
| 199 | - DataCopyExtParams copyInParams; | ||
| 200 | - copyInParams.blockCount = castDmaLoops; | ||
| 201 | - copyInParams.blockLen = VL_F32 * sizeof(M); | ||
| 202 | - copyInParams.srcStride = (VECTOR_REG_WIDTH - VL_F32 * sizeof(M)) / BLOCK_SIZE; | ||
| 203 | - copyInParams.dstStride = 0; | ||
| 204 | - DataCopyPad(meanGm[paramAddr], meanTensor.ReinterpretCast<M>(), copyInParams); | ||
| 205 | - DataCopyPad(rstdGm[paramAddr], rstdTensor.ReinterpretCast<M>(), copyInParams); | ||
| 206 | - } | ||
| 207 | - | ||
| 208 | - // tail | ||
| 209 | - uint32_t tailSize = static_cast<uint32_t>(castDmaCount % VL_F32); | ||
| 210 | - if (tailSize > 0) { | ||
| 211 | - DataCopyExtParams copyInParamsTail; | ||
| 212 | - copyInParamsTail.blockCount = 1; | ||
| 213 | - copyInParamsTail.blockLen = tailSize * sizeof(M); | ||
| 214 | - copyInParamsTail.srcStride = 0; | ||
| 215 | - copyInParamsTail.dstStride = 0; | ||
| 216 | - DataCopyPad(meanGm[paramAddr + castDmaLoops * VL_F32], | ||
| 217 | - meanTensor[castDmaLoops * VL_MEAN].ReinterpretCast<M>(), copyInParamsTail); | ||
| 218 | - DataCopyPad(rstdGm[paramAddr + castDmaLoops * VL_F32], | ||
| 219 | - rstdTensor[castDmaLoops * VL_MEAN].ReinterpretCast<M>(), copyInParamsTail); | ||
| 220 | - } | ||
| 221 | - outQueueMean.FreeTensor(meanTensor); | ||
| 222 | - outQueueRstd.FreeTensor(rstdTensor); | ||
| 223 | - } else { | ||
| 224 | - // Refresh Cache | ||
| 225 | - outQueueMean.EnQue(meanTensor); | ||
| 226 | - meanTensor = outQueueMean.template DeQue<float>(); | ||
| 227 | - CopyOut(meanGm[paramAddr], meanTensor, cacheCount); | ||
| 228 | - outQueueMean.FreeTensor(meanTensor); | ||
| 229 | - | ||
| 230 | - outQueueRstd.EnQue(rstdTensor); | ||
| 231 | - rstdTensor = outQueueRstd.template DeQue<float>(); | ||
| 232 | - CopyOut(rstdGm[paramAddr], rstdTensor, cacheCount); | ||
| 233 | - outQueueRstd.FreeTensor(rstdTensor); | ||
| 234 | - } | ||
| 235 | - paramAddr += cacheCount; | ||
| 236 | - } | ||
| 237 | - | ||
| 238 | - __aicore__ inline void WelfordInitialize(const LocalTensor<float>& mean, const LocalTensor<float>& variance, | ||
| 239 | - const int64_t elemCnt) | ||
| 240 | - { | ||
| 241 | - // WelfordInitialize | ||
| 242 | - welfordCount = 0; | ||
| 243 | - constexpr static uint32_t VL_B32 = LayerNormV4Regbase::GetVRegSize() / sizeof(float); | ||
| 244 | - uint16_t loopTimes = (elemCnt + VL_B32 - 1) / VL_B32; | ||
| 245 | - __VEC_SCOPE__ | ||
| 246 | - { | ||
| 247 | - __ubuf__ float* meamPtr = (__ubuf__ float*)mean.GetPhyAddr(); | ||
| 248 | - __ubuf__ float* variancePtr = (__ubuf__ float*)variance.GetPhyAddr(); | ||
| 249 | - uint32_t count = static_cast<uint32_t>(elemCnt); | ||
| 250 | - AscendC::Reg::RegTensor<float> xReg; | ||
| 251 | - AscendC::Reg::MaskReg pMask; | ||
| 252 | - Duplicate(xReg, 0.0f); | ||
| 253 | - for (uint16_t i = 0; i < loopTimes; ++i) { | ||
| 254 | - pMask = AscendC::Reg::UpdateMask<float>(count); | ||
| 255 | - StoreAlign((__ubuf__ float*)meamPtr + i * VL_B32, xReg, pMask); | ||
| 256 | - StoreAlign((__ubuf__ float*)variancePtr + i * VL_B32, xReg, pMask); | ||
| 257 | - } | ||
| 258 | - } | ||
| 259 | - } | ||
| 260 | - | ||
| 261 | - __aicore__ inline void ProcessWelfordUpdate(const int64_t offset, const int64_t elemCnt) | ||
| 262 | - { | ||
| 263 | - // ProcessWelfordUpdate | ||
| 264 | - welfordCount++; | ||
| 265 | - xTensor = inQueueX.template AllocTensor<T>(); | ||
| 266 | - CopyIn(xTensor, xGm[offset], elemCnt); | ||
| 267 | - inQueueX.EnQue(xTensor); | ||
| 268 | - xTensor = inQueueX.template DeQue<T>(); | ||
| 269 | - | ||
| 270 | - WelfordUpdateParam para; | ||
| 271 | - para.rnLength = 1; | ||
| 272 | - para.abLength = td_->tileLength; | ||
| 273 | - para.abComputeLength = elemCnt; | ||
| 274 | - para.nRec = 1.0f / static_cast<float>(welfordCount); | ||
| 275 | - WelfordUpdate<T, float, false>(mean_, variance_, mean_, variance_, xTensor, shared_, para); | ||
| 276 | - | ||
| 277 | - inQueueX.FreeTensor(xTensor); | ||
| 278 | - } | ||
| 279 | - | ||
| 280 | - __aicore__ inline void ProcessNormalize(const int64_t fmOffset, const int64_t paramOffset, const int64_t elemCnt) | ||
| 281 | - { | ||
| 282 | - // ProcessNormalize | ||
| 283 | - xTensor = inQueueX.template AllocTensor<T>(); | ||
| 284 | - CopyIn(xTensor, xGm[fmOffset], elemCnt); | ||
| 285 | - inQueueX.EnQue(xTensor); | ||
| 286 | - xTensor = inQueueX.template DeQue<T>(); | ||
| 287 | - | ||
| 288 | - if (td_->nullptrGamma == 0) { | ||
| 289 | - gammaTensor = inQueueGamma.template AllocTensor<U>(); | ||
| 290 | - CopyIn(gammaTensor, gammaGm[paramOffset], elemCnt); | ||
| 291 | - inQueueGamma.EnQue(gammaTensor); | ||
| 292 | - gammaTensor = inQueueGamma.template DeQue<U>(); | ||
| 293 | - } | ||
| 294 | - if (td_->nullptrBeta == 0) { | ||
| 295 | - betaTensor = inQueueBeta.template AllocTensor<U>(); | ||
| 296 | - CopyIn(betaTensor, betaGm[paramOffset], elemCnt); | ||
| 297 | - inQueueBeta.EnQue(betaTensor); | ||
| 298 | - betaTensor = inQueueBeta.template DeQue<U>(); | ||
| 299 | - } | ||
| 300 | - | ||
| 301 | - yTensor = outQueueY.template AllocTensor<T>(); | ||
| 302 | - | ||
| 303 | - NormalizePara para; | ||
| 304 | - para.aLength = 1; | ||
| 305 | - para.rLength = elemCnt; | ||
| 306 | - para.rLengthWithPadding = elemCnt; | ||
| 307 | - | ||
| 308 | - if (td_->nullptrGamma == 0 && td_->nullptrBeta == 0) { | ||
| 309 | - constexpr static NormalizeConfig hasGammaBetaNormalizeConfig = {ReducePattern::AR, -1, false, false, false}; | ||
| 310 | - Normalize<U, T, false, hasGammaBetaNormalizeConfig>(yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], | ||
| 311 | - varianceTensor[cacheCount], xTensor, gammaTensor, | ||
| 312 | - betaTensor, shared_, td_->epsilon, para); | ||
| 313 | - } else if (td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) { | ||
| 314 | - constexpr static NormalizeConfig hasGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, false, | ||
| 315 | - false}; | ||
| 316 | - Normalize<U, T, false, hasGammaNoBetaNormalizeConfig>( | ||
| 317 | - yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor, | ||
| 318 | - gammaTensor, betaTensor, shared_, td_->epsilon, para); | ||
| 319 | - } else if (!td_->nullptrGamma == 0 && td_->nullptrBeta == 0) { | ||
| 320 | - constexpr static NormalizeConfig noGammaHasBetaNormalizeConfig = {ReducePattern::AR, -1, false, true, | ||
| 321 | - false}; | ||
| 322 | - Normalize<U, T, false, noGammaHasBetaNormalizeConfig>( | ||
| 323 | - yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor, | ||
| 324 | - gammaTensor, betaTensor, shared_, td_->epsilon, para); | ||
| 325 | - } else if (!td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) { | ||
| 326 | - constexpr static NormalizeConfig noGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, true, false}; | ||
| 327 | - Normalize<U, T, false, noGammaNoBetaNormalizeConfig>( | ||
| 328 | - yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor, | ||
| 329 | - gammaTensor, betaTensor, shared_, td_->epsilon, para); | ||
| 330 | - } | ||
| 331 | - | ||
| 332 | - inQueueX.FreeTensor(xTensor); | ||
| 333 | - if (td_->nullptrGamma == 0) { | ||
| 334 | - inQueueGamma.FreeTensor(gammaTensor); | ||
| 335 | - } | ||
| 336 | - if (td_->nullptrBeta == 0) { | ||
| 337 | - inQueueBeta.FreeTensor(betaTensor); | ||
| 338 | - } | ||
| 339 | - | ||
| 340 | - outQueueY.EnQue(yTensor); | ||
| 341 | - yTensor = outQueueY.template DeQue<T>(); | ||
| 342 | - CopyOut(yGm[fmOffset], yTensor, elemCnt); | ||
| 343 | - outQueueY.FreeTensor(yTensor); | ||
| 344 | - } | ||
| 345 | - | ||
| 346 | -private: | ||
| 347 | - TPipe* pipe_; | ||
| 348 | - const LayerNormV4TilingDataWelford* __restrict td_; | ||
| 349 | - | ||
| 350 | - // Constants | ||
| 351 | - constexpr static int64_t DOUBLE_BUFFER = 2; | ||
| 352 | - constexpr static int64_t AGGREGATION_COUNT = 256; | ||
| 353 | - constexpr static uint32_t VL_F32 = VECTOR_REG_WIDTH / sizeof(float); | ||
| 354 | - constexpr static uint32_t VL_MEAN = VECTOR_REG_WIDTH / sizeof(M); | ||
| 355 | - constexpr static int64_t BLOCK_SIZE = platform::GetUbBlockSize(); | ||
| 356 | - | ||
| 357 | - // TQue | ||
| 358 | - TQue<QuePosition::VECIN, 1> inQueueX; | ||
| 359 | - TQue<QuePosition::VECIN, 1> inQueueGamma; | ||
| 360 | - TQue<QuePosition::VECIN, 1> inQueueBeta; | ||
| 361 | - | ||
| 362 | - TQue<QuePosition::VECOUT, 1> outQueueY; | ||
| 363 | - TQue<QuePosition::VECOUT, 1> outQueueMean; | ||
| 364 | - TQue<QuePosition::VECOUT, 1> outQueueRstd; | ||
| 365 | - | ||
| 366 | - TBuf<TPosition::VECCALC> welfordTempBuffer; | ||
| 367 | - TBuf<TPosition::VECCALC> apiTempBuffer; | ||
| 368 | - | ||
| 369 | - // Global Tensor | ||
| 370 | - GlobalTensor<T> xGm; | ||
| 371 | - GlobalTensor<T> yGm; | ||
| 372 | - GlobalTensor<M> meanGm; | ||
| 373 | - GlobalTensor<M> rstdGm; | ||
| 374 | - GlobalTensor<U> gammaGm; | ||
| 375 | - GlobalTensor<U> betaGm; | ||
| 376 | - | ||
| 377 | - // Local Tensor | ||
| 378 | - LocalTensor<T> xTensor; | ||
| 379 | - LocalTensor<T> yTensor; | ||
| 380 | - LocalTensor<U> gammaTensor; | ||
| 381 | - LocalTensor<U> betaTensor; | ||
| 382 | - LocalTensor<float> meanTensor; | ||
| 383 | - LocalTensor<float> rstdTensor; | ||
| 384 | - LocalTensor<float> varianceTensor; | ||
| 385 | - LocalTensor<float> mean_; | ||
| 386 | - LocalTensor<float> variance_; | ||
| 387 | - LocalTensor<uint8_t> shared_; | ||
| 388 | - | ||
| 389 | - // params | ||
| 390 | int64_t currentBlockFactor = 0; | 116 | int64_t currentBlockFactor = 0; |
| 391 | - int64_t cacheCount = 0; | ||
| 392 | - int64_t paramAddr = 0; | ||
| 393 | - int64_t welfordCount = 0; | ||
| 394 | }; | 117 | }; |
| 395 | 118 | ||
| 396 | } // namespace LayerNormV4 | 119 | } // namespace LayerNormV4 |
| @@ -0,0 +1,317 @@ | |||
| 1 | +/** | ||
| 2 | + * Copyright (c) 2026 Huawei Technologies Co., Ltd. | ||
| 3 | + * This program is free software, you can redistribute it and/or modify it under the terms and conditions of | ||
| 4 | + * CANN Open Software License Agreement Version 2.0 (the "License"). | ||
| 5 | + * Please refer to the License for details. You may not use this file except in compliance with the License. | ||
| 6 | + * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, | ||
| 7 | + * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. | ||
| 8 | + * See LICENSE in the root of the software repository for the full text of the License. | ||
| 9 | + */ | ||
| 10 | + | ||
| 11 | +/*! | ||
| 12 | + * \file layer_norm_v4_welford_base.h | ||
| 13 | + * \brief | ||
| 14 | + */ | ||
| 15 | + | ||
| 16 | +#ifndef LAYER_NORM_V4_WELFORD_BASE_H | ||
| 17 | +#define LAYER_NORM_V4_WELFORD_BASE_H | ||
| 18 | + | ||
| 19 | +#include "layer_norm_v4_regbase_common.h" | ||
| 20 | +#include "kernel_tiling/kernel_tiling.h" | ||
| 21 | +#include "kernel_operator.h" | ||
| 22 | +#include "../../inc/platform.h" | ||
| 23 | + | ||
| 24 | +namespace LayerNormV4 { | ||
| 25 | +using namespace AscendC; | ||
| 26 | +using AscendC::Reg::StoreAlign; | ||
| 27 | + | ||
| 28 | +namespace LayerNormV4Regbase { | ||
| 29 | +__aicore__ inline constexpr uint32_t GetVRegSize() | ||
| 30 | +{ | ||
| 31 | +#if __CCE_AICORE__ == 310 || __CCE_AICORE__ == 920 || __NPU_ARCH == 5102 | ||
| 32 | + return AscendC::VECTOR_REG_WIDTH; | ||
| 33 | +#else | ||
| 34 | + return 256U; | ||
| 35 | +#endif | ||
| 36 | +} | ||
| 37 | +} // namespace LayerNormV4Regbase | ||
| 38 | + | ||
| 39 | +template <typename T> | ||
| 40 | +__aicore__ inline void CopyIn(const LocalTensor<T>& dstTensor, const GlobalTensor<T>& srcTensor, const int64_t rowSize) | ||
| 41 | +{ | ||
| 42 | + // CopyIn | ||
| 43 | + DataCopyExtParams params; | ||
| 44 | + params.blockCount = 1; | ||
| 45 | + params.blockLen = rowSize * sizeof(T); | ||
| 46 | + DataCopyPadExtParams<T> padParams; | ||
| 47 | + padParams.isPad = false; | ||
| 48 | + DataCopyPad(dstTensor, srcTensor, params, padParams); | ||
| 49 | +} | ||
| 50 | + | ||
| 51 | +template <typename T> | ||
| 52 | +__aicore__ inline void CopyOut(const GlobalTensor<T>& dstTensor, const LocalTensor<T>& srcTensor, const int64_t rowSize) | ||
| 53 | +{ | ||
| 54 | + // CopyOut | ||
| 55 | + DataCopyExtParams params; | ||
| 56 | + params.blockCount = 1; | ||
| 57 | + params.blockLen = rowSize * sizeof(T); | ||
| 58 | + DataCopyPad(dstTensor, srcTensor, params); | ||
| 59 | +} | ||
| 60 | + | ||
| 61 | +// Shared buffers and tile computation; execution scheduling belongs to derived kernels. | ||
| 62 | +template <typename T, typename U, typename M, typename TilingData> | ||
| 63 | +class LayerNormV4WelfordBase { | ||
| 64 | +protected: | ||
| 65 | + __aicore__ inline LayerNormV4WelfordBase(const TilingData* tilingData, TPipe* pipeIn) | ||
| 66 | + : pipe_(pipeIn), td_(tilingData) | ||
| 67 | + {} | ||
| 68 | + | ||
| 69 | + __aicore__ inline void InitCommon(GM_ADDR gamma, GM_ADDR beta) | ||
| 70 | + { | ||
| 71 | + if (td_->nullptrGamma == 0) { | ||
| 72 | + gammaGm.SetGlobalBuffer((__gm__ U*)gamma, td_->N); | ||
| 73 | + } | ||
| 74 | + if (td_->nullptrBeta == 0) { | ||
| 75 | + betaGm.SetGlobalBuffer((__gm__ U*)beta, td_->N); | ||
| 76 | + } | ||
| 77 | + | ||
| 78 | + // init local memory | ||
| 79 | + pipe_->InitBuffer(inQueueX, DOUBLE_BUFFER, td_->tileLength * sizeof(T)); | ||
| 80 | + pipe_->InitBuffer(outQueueY, DOUBLE_BUFFER, td_->tileLength * sizeof(T)); | ||
| 81 | + pipe_->InitBuffer(outQueueMean, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float)); | ||
| 82 | + pipe_->InitBuffer(outQueueRstd, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float)); | ||
| 83 | + pipe_->InitBuffer(welfordTempBuffer, td_->welfordTempSize); | ||
| 84 | + if (td_->nullptrGamma == 0) { | ||
| 85 | + pipe_->InitBuffer(inQueueGamma, DOUBLE_BUFFER, td_->tileLength * sizeof(U)); | ||
| 86 | + } | ||
| 87 | + if (td_->nullptrBeta == 0) { | ||
| 88 | + pipe_->InitBuffer(inQueueBeta, DOUBLE_BUFFER, td_->tileLength * sizeof(U)); | ||
| 89 | + } | ||
| 90 | + pipe_->InitBuffer(apiTempBuffer, td_->apiTempBufferSize); | ||
| 91 | + } | ||
| 92 | + | ||
| 93 | + __aicore__ inline void ResetCache() | ||
| 94 | + { | ||
| 95 | + // ResetCache | ||
| 96 | + cacheCount = 0; | ||
| 97 | + meanTensor = outQueueMean.template AllocTensor<float>(); | ||
| 98 | + rstdTensor = outQueueRstd.template AllocTensor<float>(); | ||
| 99 | + } | ||
| 100 | + | ||
| 101 | + __aicore__ inline void RefreshCache() | ||
| 102 | + { | ||
| 103 | + if constexpr (!IsSameType<M, float>::value) { | ||
| 104 | + // float to bfloat16 or float16, input continue and output each repeat have only half value | ||
| 105 | + CastBatchMeanRstdToDtype<M>((__ubuf__ float*)meanTensor.GetPhyAddr(), | ||
| 106 | + (__ubuf__ float*)rstdTensor.GetPhyAddr(), (__ubuf__ M*)meanTensor.GetPhyAddr(), | ||
| 107 | + (__ubuf__ M*)rstdTensor.GetPhyAddr(), cacheCount); | ||
| 108 | + outQueueMean.EnQue(meanTensor); | ||
| 109 | + outQueueRstd.EnQue(rstdTensor); | ||
| 110 | + meanTensor = outQueueMean.template DeQue<float>(); | ||
| 111 | + rstdTensor = outQueueRstd.template DeQue<float>(); | ||
| 112 | + | ||
| 113 | + // VL_F32 | ||
| 114 | + uint32_t castDmaCount = static_cast<uint32_t>(cacheCount); | ||
| 115 | + uint32_t castDmaLoops = static_cast<uint32_t>(castDmaCount / VL_F32); | ||
| 116 | + if (castDmaLoops > 0) { | ||
| 117 | + DataCopyExtParams copyInParams; | ||
| 118 | + copyInParams.blockCount = castDmaLoops; | ||
| 119 | + copyInParams.blockLen = VL_F32 * sizeof(M); | ||
| 120 | + copyInParams.srcStride = (VECTOR_REG_WIDTH - VL_F32 * sizeof(M)) / BLOCK_SIZE; | ||
| 121 | + copyInParams.dstStride = 0; | ||
| 122 | + DataCopyPad(meanGm[paramAddr], meanTensor.ReinterpretCast<M>(), copyInParams); | ||
| 123 | + DataCopyPad(rstdGm[paramAddr], rstdTensor.ReinterpretCast<M>(), copyInParams); | ||
| 124 | + } | ||
| 125 | + | ||
| 126 | + // tail | ||
| 127 | + uint32_t tailSize = static_cast<uint32_t>(castDmaCount % VL_F32); | ||
| 128 | + if (tailSize > 0) { | ||
| 129 | + DataCopyExtParams copyInParamsTail; | ||
| 130 | + copyInParamsTail.blockCount = 1; | ||
| 131 | + copyInParamsTail.blockLen = tailSize * sizeof(M); | ||
| 132 | + copyInParamsTail.srcStride = 0; | ||
| 133 | + copyInParamsTail.dstStride = 0; | ||
| 134 | + // meanTensor is a LocalTensor<float> and the cast output packs each | ||
| 135 | + // repeat slot at VECTOR_REG_WIDTH bytes, so the tail must be indexed in | ||
| 136 | + // float units (castDmaLoops * VL_F32), matching the GM offset above. | ||
| 137 | + DataCopyPad(meanGm[paramAddr + castDmaLoops * VL_F32], | ||
| 138 | + meanTensor[castDmaLoops * VL_F32].ReinterpretCast<M>(), copyInParamsTail); | ||
| 139 | + DataCopyPad(rstdGm[paramAddr + castDmaLoops * VL_F32], | ||
| 140 | + rstdTensor[castDmaLoops * VL_F32].ReinterpretCast<M>(), copyInParamsTail); | ||
| 141 | + } | ||
| 142 | + outQueueMean.FreeTensor(meanTensor); | ||
| 143 | + outQueueRstd.FreeTensor(rstdTensor); | ||
| 144 | + } else { | ||
| 145 | + // Refresh Cache | ||
| 146 | + outQueueMean.EnQue(meanTensor); | ||
| 147 | + meanTensor = outQueueMean.template DeQue<float>(); | ||
| 148 | + CopyOut(meanGm[paramAddr], meanTensor, cacheCount); | ||
| 149 | + outQueueMean.FreeTensor(meanTensor); | ||
| 150 | + | ||
| 151 | + outQueueRstd.EnQue(rstdTensor); | ||
| 152 | + rstdTensor = outQueueRstd.template DeQue<float>(); | ||
| 153 | + CopyOut(rstdGm[paramAddr], rstdTensor, cacheCount); | ||
| 154 | + outQueueRstd.FreeTensor(rstdTensor); | ||
| 155 | + } | ||
| 156 | + paramAddr += cacheCount; | ||
| 157 | + } | ||
| 158 | + | ||
| 159 | + __aicore__ inline void WelfordInitialize(const LocalTensor<float>& mean, const LocalTensor<float>& variance, | ||
| 160 | + const int64_t elemCnt) | ||
| 161 | + { | ||
| 162 | + // WelfordInitialize | ||
| 163 | + welfordCount = 0; | ||
| 164 | + constexpr static uint32_t VL_B32 = LayerNormV4Regbase::GetVRegSize() / sizeof(float); | ||
| 165 | + uint16_t loopTimes = (elemCnt + VL_B32 - 1) / VL_B32; | ||
| 166 | + __VEC_SCOPE__ | ||
| 167 | + { | ||
| 168 | + __ubuf__ float* meamPtr = (__ubuf__ float*)mean.GetPhyAddr(); | ||
| 169 | + __ubuf__ float* variancePtr = (__ubuf__ float*)variance.GetPhyAddr(); | ||
| 170 | + uint32_t count = static_cast<uint32_t>(elemCnt); | ||
| 171 | + AscendC::Reg::RegTensor<float> xReg; | ||
| 172 | + AscendC::Reg::MaskReg pMask; | ||
| 173 | + Duplicate(xReg, 0.0f); | ||
| 174 | + for (uint16_t i = 0; i < loopTimes; ++i) { | ||
| 175 | + pMask = AscendC::Reg::UpdateMask<float>(count); | ||
| 176 | + StoreAlign((__ubuf__ float*)meamPtr + i * VL_B32, xReg, pMask); | ||
| 177 | + StoreAlign((__ubuf__ float*)variancePtr + i * VL_B32, xReg, pMask); | ||
| 178 | + } | ||
| 179 | + } | ||
| 180 | + } | ||
| 181 | + | ||
| 182 | + __aicore__ inline void ProcessWelfordUpdate(const int64_t offset, const int64_t elemCnt) | ||
| 183 | + { | ||
| 184 | + // ProcessWelfordUpdate | ||
| 185 | + welfordCount++; | ||
| 186 | + xTensor = inQueueX.template AllocTensor<T>(); | ||
| 187 | + CopyIn(xTensor, xGm[offset], elemCnt); | ||
| 188 | + inQueueX.EnQue(xTensor); | ||
| 189 | + xTensor = inQueueX.template DeQue<T>(); | ||
| 190 | + | ||
| 191 | + WelfordUpdateParam para; | ||
| 192 | + para.rnLength = 1; | ||
| 193 | + para.abLength = td_->tileLength; | ||
| 194 | + para.abComputeLength = elemCnt; | ||
| 195 | + para.nRec = 1.0f / static_cast<float>(welfordCount); | ||
| 196 | + WelfordUpdate<T, float, false>(mean_, variance_, mean_, variance_, xTensor, shared_, para); | ||
| 197 | + | ||
| 198 | + inQueueX.FreeTensor(xTensor); | ||
| 199 | + } | ||
| 200 | + | ||
| 201 | + __aicore__ inline void ProcessNormalize(const int64_t fmOffset, const int64_t paramOffset, const int64_t elemCnt) | ||
| 202 | + { | ||
| 203 | + // ProcessNormalize | ||
| 204 | + xTensor = inQueueX.template AllocTensor<T>(); | ||
| 205 | + CopyIn(xTensor, xGm[fmOffset], elemCnt); | ||
| 206 | + inQueueX.EnQue(xTensor); | ||
| 207 | + xTensor = inQueueX.template DeQue<T>(); | ||
| 208 | + | ||
| 209 | + if (td_->nullptrGamma == 0) { | ||
| 210 | + gammaTensor = inQueueGamma.template AllocTensor<U>(); | ||
| 211 | + CopyIn(gammaTensor, gammaGm[paramOffset], elemCnt); | ||
| 212 | + inQueueGamma.EnQue(gammaTensor); | ||
| 213 | + gammaTensor = inQueueGamma.template DeQue<U>(); | ||
| 214 | + } | ||
| 215 | + if (td_->nullptrBeta == 0) { | ||
| 216 | + betaTensor = inQueueBeta.template AllocTensor<U>(); | ||
| 217 | + CopyIn(betaTensor, betaGm[paramOffset], elemCnt); | ||
| 218 | + inQueueBeta.EnQue(betaTensor); | ||
| 219 | + betaTensor = inQueueBeta.template DeQue<U>(); | ||
| 220 | + } | ||
| 221 | + | ||
| 222 | + yTensor = outQueueY.template AllocTensor<T>(); | ||
| 223 | + | ||
| 224 | + NormalizePara para; | ||
| 225 | + para.aLength = 1; | ||
| 226 | + para.rLength = elemCnt; | ||
| 227 | + para.rLengthWithPadding = elemCnt; | ||
| 228 | + | ||
| 229 | + if (td_->nullptrGamma == 0 && td_->nullptrBeta == 0) { | ||
| 230 | + constexpr static NormalizeConfig hasGammaBetaNormalizeConfig = {ReducePattern::AR, -1, false, false, false}; | ||
| 231 | + Normalize<U, T, false, hasGammaBetaNormalizeConfig>(yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], | ||
| 232 | + varianceTensor[cacheCount], xTensor, gammaTensor, | ||
| 233 | + betaTensor, shared_, td_->epsilon, para); | ||
| 234 | + } else if (td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) { | ||
| 235 | + constexpr static NormalizeConfig hasGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, false, | ||
| 236 | + false}; | ||
| 237 | + Normalize<U, T, false, hasGammaNoBetaNormalizeConfig>( | ||
| 238 | + yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor, | ||
| 239 | + gammaTensor, betaTensor, shared_, td_->epsilon, para); | ||
| 240 | + } else if (!td_->nullptrGamma == 0 && td_->nullptrBeta == 0) { | ||
| 241 | + constexpr static NormalizeConfig noGammaHasBetaNormalizeConfig = {ReducePattern::AR, -1, false, true, | ||
| 242 | + false}; | ||
| 243 | + Normalize<U, T, false, noGammaHasBetaNormalizeConfig>( | ||
| 244 | + yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor, | ||
| 245 | + gammaTensor, betaTensor, shared_, td_->epsilon, para); | ||
| 246 | + } else if (!td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) { | ||
| 247 | + constexpr static NormalizeConfig noGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, true, false}; | ||
| 248 | + Normalize<U, T, false, noGammaNoBetaNormalizeConfig>( | ||
| 249 | + yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor, | ||
| 250 | + gammaTensor, betaTensor, shared_, td_->epsilon, para); | ||
| 251 | + } | ||
| 252 | + | ||
| 253 | + inQueueX.FreeTensor(xTensor); | ||
| 254 | + if (td_->nullptrGamma == 0) { | ||
| 255 | + inQueueGamma.FreeTensor(gammaTensor); | ||
| 256 | + } | ||
| 257 | + if (td_->nullptrBeta == 0) { | ||
| 258 | + inQueueBeta.FreeTensor(betaTensor); | ||
| 259 | + } | ||
| 260 | + | ||
| 261 | + outQueueY.EnQue(yTensor); | ||
| 262 | + yTensor = outQueueY.template DeQue<T>(); | ||
| 263 | + CopyOut(yGm[fmOffset], yTensor, elemCnt); | ||
| 264 | + outQueueY.FreeTensor(yTensor); | ||
| 265 | + } | ||
| 266 | + | ||
| 267 | + TPipe* pipe_; | ||
| 268 | + const TilingData* __restrict td_; | ||
| 269 | + | ||
| 270 | + // Constants | ||
| 271 | + constexpr static int64_t DOUBLE_BUFFER = 2; | ||
| 272 | + constexpr static int64_t AGGREGATION_COUNT = 256; | ||
| 273 | + constexpr static uint32_t VL_F32 = VECTOR_REG_WIDTH / sizeof(float); | ||
| 274 | + constexpr static uint32_t VL_MEAN = VECTOR_REG_WIDTH / sizeof(M); | ||
| 275 | + constexpr static int64_t BLOCK_SIZE = platform::GetUbBlockSize(); | ||
| 276 | + | ||
| 277 | + // TQue | ||
| 278 | + TQue<QuePosition::VECIN, 1> inQueueX; | ||
| 279 | + TQue<QuePosition::VECIN, 1> inQueueGamma; | ||
| 280 | + TQue<QuePosition::VECIN, 1> inQueueBeta; | ||
| 281 | + | ||
| 282 | + TQue<QuePosition::VECOUT, 1> outQueueY; | ||
| 283 | + TQue<QuePosition::VECOUT, 1> outQueueMean; | ||
| 284 | + TQue<QuePosition::VECOUT, 1> outQueueRstd; | ||
| 285 | + | ||
| 286 | + TBuf<TPosition::VECCALC> welfordTempBuffer; | ||
| 287 | + TBuf<TPosition::VECCALC> apiTempBuffer; | ||
| 288 | + | ||
| 289 | + // Global Tensor | ||
| 290 | + GlobalTensor<T> xGm; | ||
| 291 | + GlobalTensor<T> yGm; | ||
| 292 | + GlobalTensor<M> meanGm; | ||
| 293 | + GlobalTensor<M> rstdGm; | ||
| 294 | + GlobalTensor<U> gammaGm; | ||
| 295 | + GlobalTensor<U> betaGm; | ||
| 296 | + | ||
| 297 | + // Local Tensor | ||
| 298 | + LocalTensor<T> xTensor; | ||
| 299 | + LocalTensor<T> yTensor; | ||
| 300 | + LocalTensor<U> gammaTensor; | ||
| 301 | + LocalTensor<U> betaTensor; | ||
| 302 | + LocalTensor<float> meanTensor; | ||
| 303 | + LocalTensor<float> rstdTensor; | ||
| 304 | + LocalTensor<float> varianceTensor; | ||
| 305 | + LocalTensor<float> mean_; | ||
| 306 | + LocalTensor<float> variance_; | ||
| 307 | + LocalTensor<uint8_t> shared_; | ||
| 308 | + | ||
| 309 | + // params | ||
| 310 | + int64_t cacheCount = 0; | ||
| 311 | + int64_t paramAddr = 0; | ||
| 312 | + int64_t welfordCount = 0; | ||
| 313 | +}; | ||
| 314 | + | ||
| 315 | +} // namespace LayerNormV4 | ||
| 316 | + | ||
| 317 | +#endif // LAYER_NORM_V4_WELFORD_BASE_H | ||
| @@ -19,6 +19,7 @@ | |||
| 19 | #include "kernel_operator.h" | 19 | #include "kernel_operator.h" |
| 20 | #include "arch35/layer_norm_v4_two_pass.h" | 20 | #include "arch35/layer_norm_v4_two_pass.h" |
| 21 | #include "arch35/layer_norm_v4_welford.h" | 21 | #include "arch35/layer_norm_v4_welford.h" |
| 22 | +#include "arch35/layer_norm_v4_split_r.h" | ||
| 22 | #include "arch35/layer_norm_v4_two_pass_perf.h" | 23 | #include "arch35/layer_norm_v4_two_pass_perf.h" |
| 23 | 24 | ||
| 24 | using namespace LayerNormV4; | 25 | using namespace LayerNormV4; |
| @@ -41,6 +42,12 @@ using AscendC::AIC; | |||
| 41 | #define LNV4_REGBASE_TWO_PASS_PERF_BF16_FLOAT 520 | 42 | #define LNV4_REGBASE_TWO_PASS_PERF_BF16_FLOAT 520 |
| 42 | #define LNV4_REGBASE_TWO_PASS_PERF_BF16_BF16 522 | 43 | #define LNV4_REGBASE_TWO_PASS_PERF_BF16_BF16 522 |
| 43 | 44 | ||
| 45 | +#define LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT 800 | ||
| 46 | +#define LNV4_REGBASE_SPLIT_R_HALF_FLOAT 810 | ||
| 47 | +#define LNV4_REGBASE_SPLIT_R_HALF_HALF 811 | ||
| 48 | +#define LNV4_REGBASE_SPLIT_R_BF16_FLOAT 820 | ||
| 49 | +#define LNV4_REGBASE_SPLIT_R_BF16_BF16 822 | ||
| 50 | + | ||
| 44 | template <typename Tfm, typename Tweight, typename Tmean> | 51 | template <typename Tfm, typename Tweight, typename Tmean> |
| 45 | __aicore__ inline void RegbaseTwoPassImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd, | 52 | __aicore__ inline void RegbaseTwoPassImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd, |
| 46 | GM_ADDR tiling) | 53 | GM_ADDR tiling) |
| @@ -76,6 +83,24 @@ __aicore__ inline void RegbaseTwoPassPerfImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR | |||
| 76 | op.Process(); | 83 | op.Process(); |
| 77 | } | 84 | } |
| 78 | 85 | ||
| 86 | +template <typename T, typename U> | ||
| 87 | +__aicore__ inline void RegbaseSplitRImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd, | ||
| 88 | + GM_ADDR workspace, GM_ADDR tiling) | ||
| 89 | +{ | ||
| 90 | + if (workspace == nullptr) { | ||
| 91 | + return; | ||
| 92 | + } | ||
| 93 | + GM_ADDR userWS = GetUserWorkspace(workspace); | ||
| 94 | + if (userWS == nullptr) { | ||
| 95 | + return; | ||
| 96 | + } | ||
| 97 | + TPipe pipe; | ||
| 98 | + GET_TILING_DATA_WITH_STRUCT(LayerNormV4TilingDataSplitR, tilingData, tiling); | ||
| 99 | + LayerNormV4RegbaseSplitR<T, U, DTYPE_MEAN> op(&tilingData, &pipe); | ||
| 100 | + op.Init(x, gamma, beta, y, mean, rstd, userWS); | ||
| 101 | + op.Process(); | ||
| 102 | +} | ||
| 103 | + | ||
| 79 | static __aicore__ inline void layer_norm_impl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, | 104 | static __aicore__ inline void layer_norm_impl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, |
| 80 | GM_ADDR rstd, GM_ADDR workspace, GM_ADDR tiling) | 105 | GM_ADDR rstd, GM_ADDR workspace, GM_ADDR tiling) |
| 81 | { | 106 | { |
| @@ -83,6 +108,32 @@ static __aicore__ inline void layer_norm_impl(GM_ADDR x, GM_ADDR gamma, GM_ADDR | |||
| 83 | return; | 108 | return; |
| 84 | } | 109 | } |
| 85 | KERNEL_TASK_TYPE_DEFAULT(KERNEL_TYPE_AIV_ONLY); | 110 | KERNEL_TASK_TYPE_DEFAULT(KERNEL_TYPE_AIV_ONLY); |
| 111 | + // Hardware SyncAll requires MIX scheduling even when only AIV cores execute. | ||
| 112 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT)) { | ||
| 113 | + KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT, KERNEL_TYPE_MIX_AIV_1_0); | ||
| 114 | + RegbaseSplitRImpl<float, float>(x, gamma, beta, y, mean, rstd, workspace, tiling); | ||
| 115 | + return; | ||
| 116 | + } | ||
| 117 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_HALF_FLOAT)) { | ||
| 118 | + KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_HALF_FLOAT, KERNEL_TYPE_MIX_AIV_1_0); | ||
| 119 | + RegbaseSplitRImpl<half, float>(x, gamma, beta, y, mean, rstd, workspace, tiling); | ||
| 120 | + return; | ||
| 121 | + } | ||
| 122 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_HALF_HALF)) { | ||
| 123 | + KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_HALF_HALF, KERNEL_TYPE_MIX_AIV_1_0); | ||
| 124 | + RegbaseSplitRImpl<half, half>(x, gamma, beta, y, mean, rstd, workspace, tiling); | ||
| 125 | + return; | ||
| 126 | + } | ||
| 127 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_BF16_FLOAT)) { | ||
| 128 | + KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_BF16_FLOAT, KERNEL_TYPE_MIX_AIV_1_0); | ||
| 129 | + RegbaseSplitRImpl<bfloat16_t, float>(x, gamma, beta, y, mean, rstd, workspace, tiling); | ||
| 130 | + return; | ||
| 131 | + } | ||
| 132 | + if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_BF16_BF16)) { | ||
| 133 | + KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_BF16_BF16, KERNEL_TYPE_MIX_AIV_1_0); | ||
| 134 | + RegbaseSplitRImpl<bfloat16_t, bfloat16_t>(x, gamma, beta, y, mean, rstd, workspace, tiling); | ||
| 135 | + return; | ||
| 136 | + } | ||
| 86 | if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) { | 137 | if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) { |
| 87 | RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(x, gamma, beta, y, mean, rstd, tiling); | 138 | RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(x, gamma, beta, y, mean, rstd, tiling); |
| 88 | return; | 139 | return; |
| @@ -21,3 +21,54 @@ LayerNormV4_float16_none_5,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), None, None) | |||
| 21 | LayerNormV4_float16_none_6,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),() | 21 | LayerNormV4_float16_none_6,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),() |
| 22 | LayerNormV4_float16_none_7,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), None, (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), None, (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),() | 22 | LayerNormV4_float16_none_7,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), None, (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), None, (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),() |
| 23 | LayerNormV4_float16_none_8,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), None)","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),() | 23 | LayerNormV4_float16_none_8,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), None)","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),() |
| 24 | +LayerNormV4_fallback_float32_float32_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),() | ||
| 25 | +LayerNormV4_split_r_float32_float32_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 26 | +LayerNormV4_split_r_float32_float32_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 27 | +LayerNormV4_split_r_float32_float32_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 28 | +LayerNormV4_fallback_float16_float16_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),() | ||
| 29 | +LayerNormV4_split_r_float16_float16_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 30 | +LayerNormV4_split_r_float16_float16_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 31 | +LayerNormV4_split_r_float16_float16_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 32 | +LayerNormV4_fallback_float16_float32_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),() | ||
| 33 | +LayerNormV4_split_r_float16_float32_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 34 | +LayerNormV4_split_r_float16_float32_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 35 | +LayerNormV4_split_r_float16_float32_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 36 | +LayerNormV4_fallback_bfloat16_bfloat16_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),() | ||
| 37 | +LayerNormV4_split_r_bfloat16_bfloat16_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 38 | +LayerNormV4_split_r_bfloat16_bfloat16_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 39 | +LayerNormV4_split_r_bfloat16_bfloat16_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 40 | +LayerNormV4_fallback_bfloat16_float32_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),() | ||
| 41 | +LayerNormV4_split_r_bfloat16_float32_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 42 | +LayerNormV4_split_r_bfloat16_float32_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 43 | +LayerNormV4_split_r_bfloat16_float32_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),() | ||
| 44 | +LayerNormV4_split_r_float32_zero,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((0, 0), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 45 | +LayerNormV4_split_r_float32_constant,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((7, 7), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 46 | +LayerNormV4_split_r_float32_large_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1002), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 47 | +LayerNormV4_split_r_float16_zero,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((0, 0), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 48 | +LayerNormV4_split_r_float16_constant,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((7, 7), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 49 | +LayerNormV4_split_r_float16_large_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1002), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 50 | +LayerNormV4_split_r_bfloat16_zero,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((0, 0), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 51 | +LayerNormV4_split_r_bfloat16_constant,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((7, 7), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 52 | +LayerNormV4_split_r_bfloat16_large_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1002), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),() | ||
| 53 | +LayerNormV4_split_r_float32_no_gamma,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), None, (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), None, (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR可选参数缺省,,0,,(),() | ||
| 54 | +LayerNormV4_split_r_float32_no_beta,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), None)","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR可选参数缺省,,0,,(),() | ||
| 55 | +LayerNormV4_split_r_float32_no_params,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), None, None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), None, None)","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR可选参数缺省,,0,,(),() | ||
| 56 | +LayerNormV4_split_r_float32_threshold_65535,UNKNOWN,layer_norm_v4,"((1, 65535), (1,), (65535,), (65535,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 65535), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65535), (1,), (65535,), (65535,))","('ND', 'ND', 'ND', 'ND')","((1, 65535), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR准入阈值边界,,0,,(),() | ||
| 57 | +LayerNormV4_split_r_float32_threshold_65536,UNKNOWN,layer_norm_v4,"((1, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR准入阈值边界,,0,,(),() | ||
| 58 | +LayerNormV4_split_r_asweep_float16_8_65536,UNKNOWN,layer_norm_v4,"((8, 65536), (1,), (65536,), (65536,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 59 | +LayerNormV4_split_r_asweep_float16_8_262145,UNKNOWN,layer_norm_v4,"((8, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 60 | +LayerNormV4_split_r_asweep_float16_16_65536,UNKNOWN,layer_norm_v4,"((16, 65536), (1,), (65536,), (65536,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 61 | +LayerNormV4_split_r_asweep_float16_16_262145,UNKNOWN,layer_norm_v4,"((16, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 62 | +LayerNormV4_split_r_asweep_float16_32_65536,UNKNOWN,layer_norm_v4,"((32, 65536), (1,), (65536,), (65536,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 63 | +LayerNormV4_split_r_asweep_float16_32_262145,UNKNOWN,layer_norm_v4,"((32, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 64 | +LayerNormV4_split_r_asweep_float32_8_65536,UNKNOWN,layer_norm_v4,"((8, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 65 | +LayerNormV4_split_r_asweep_float32_8_262145,UNKNOWN,layer_norm_v4,"((8, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 66 | +LayerNormV4_split_r_asweep_float32_16_65536,UNKNOWN,layer_norm_v4,"((16, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 67 | +LayerNormV4_split_r_asweep_float32_16_262145,UNKNOWN,layer_norm_v4,"((16, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 68 | +LayerNormV4_split_r_asweep_float32_32_65536,UNKNOWN,layer_norm_v4,"((32, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 69 | +LayerNormV4_split_r_asweep_float32_32_262145,UNKNOWN,layer_norm_v4,"((32, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),() | ||
| 70 | +LayerNormV4_split_r_bfloat16_representable_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1016), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,BF16在1000附近ULP约4,使用可表示扰动验证小方差,,0,,(),() | ||
| 71 | +LayerNormV4_split_r_float32_finite_variance_65536,UNKNOWN,layer_norm_v4,"((1, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),() | ||
| 72 | +LayerNormV4_split_r_float32_finite_variance_81921,UNKNOWN,layer_norm_v4,"((1, 81921), (1,), (81921,), (81921,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 81921), (1,), (81921,), (81921,))","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),() | ||
| 73 | +LayerNormV4_split_r_bfloat16_finite_variance_65536,UNKNOWN,layer_norm_v4,"((1, 65536), (1,), (65536,), (65536,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),() | ||
| 74 | +LayerNormV4_split_r_bfloat16_finite_variance_81921,UNKNOWN,layer_norm_v4,"((1, 81921), (1,), (81921,), (81921,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 81921), (1,), (81921,), (81921,))","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),() | ||
| @@ -14,6 +14,7 @@ | |||
| 14 | */ | 14 | */ |
| 15 | 15 | ||
| 16 | #include <iostream> | 16 | #include <iostream> |
| 17 | +#include <cstring> | ||
| 17 | #include <fstream> | 18 | #include <fstream> |
| 18 | #include <vector> | 19 | #include <vector> |
| 19 | #include <gtest/gtest.h> | 20 | #include <gtest/gtest.h> |
| @@ -25,6 +26,8 @@ | |||
| 25 | #include "kernel_run_context_facker.h" | 26 | #include "kernel_run_context_facker.h" |
| 26 | #include "test_cube_util.h" | 27 | #include "test_cube_util.h" |
| 27 | #include "base/err_mgr.h" | 28 | #include "base/err_mgr.h" |
| 29 | +#include "tiling/platform/platform_ascendc.h" | ||
| 30 | +#include "op_common/op_host/util/platform_util.h" | ||
| 28 | 31 | ||
| 29 | using namespace ut_util; | 32 | using namespace ut_util; |
| 30 | using namespace std; | 33 | using namespace std; |
| @@ -2011,3 +2014,159 @@ static const auto _inst_LAYER_NORM_V4 = []() { | |||
| 2011 | testing::UnitTest::GetInstance()->listeners().Append(new LAYER_NORM_V4NoEz9999Check()); | 2014 | testing::UnitTest::GetInstance()->listeners().Append(new LAYER_NORM_V4NoEz9999Check()); |
| 2012 | return 0; | 2015 | return 0; |
| 2013 | }(); | 2016 | }(); |
| 2017 | + | ||
| 2018 | +// Prevent regressions in SplitR dispatch and fallback at the shape boundaries. | ||
| 2019 | +static void CheckLayerNormV4SplitR(int64_t a, int64_t r, int64_t expectedSplits) | ||
| 2020 | +{ | ||
| 2021 | + gert::StorageShape x_shape = {{a, r}, {a, r}}; | ||
| 2022 | + gert::StorageShape normalized_shape = {{1}, {1}}; | ||
| 2023 | + gert::StorageShape gamma_shape = {{r}, {r}}; | ||
| 2024 | + gert::StorageShape beta_shape = {{r}, {r}}; | ||
| 2025 | + gert::StorageShape y_shape = {{a, r}, {a, r}}; | ||
| 2026 | + gert::StorageShape mean_shape = {{a}, {a}}; | ||
| 2027 | + gert::StorageShape rstd_shape = {{a}, {a}}; | ||
| 2028 | + float eps = 1e-5; | ||
| 2029 | + | ||
| 2030 | + string compile_info_string = R"({ | ||
| 2031 | + "hardware_info": {"BT_SIZE": 0, "load3d_constraints": "1", | ||
| 2032 | + "Intrinsic_fix_pipe_l0c2out": false, | ||
| 2033 | + "Intrinsic_data_move_l12ub": true, | ||
| 2034 | + "Intrinsic_data_move_l0c2ub": true, | ||
| 2035 | + "Intrinsic_data_move_out2l1_nd2nz": false, | ||
| 2036 | + "UB_SIZE": 245760, "L2_SIZE": 33554432, "L1_SIZE": 524288, | ||
| 2037 | + "L0A_SIZE": 65536, "L0B_SIZE": 65536, "L0C_SIZE": 131072, | ||
| 2038 | + "CORE_NUM": 64, "socVersion": "Ascend950"} | ||
| 2039 | + })"; | ||
| 2040 | + map<string, string> soc_infos; | ||
| 2041 | + map<string, string> aicore_spec; | ||
| 2042 | + map<string, string> intrinsics; | ||
| 2043 | + map<string, string> soc_version; | ||
| 2044 | + map<string, string> npuarchs = {{"NpuArch", "3510"}}; | ||
| 2045 | + GetPlatFormInfos(compile_info_string.c_str(), soc_infos, aicore_spec, intrinsics, soc_version); | ||
| 2046 | + | ||
| 2047 | + fe::PlatFormInfos platform_info; | ||
| 2048 | + platform_info.Init(); | ||
| 2049 | + LayerNormV4CompileInfo compile_info; | ||
| 2050 | + | ||
| 2051 | + std::string op_type("LayerNormV4"); | ||
| 2052 | + ASSERT_NE(gert::OpImplRegistry::GetInstance().GetOpImpl(op_type.c_str()), nullptr); | ||
| 2053 | + auto tiling_func = gert::OpImplRegistry::GetInstance().GetOpImpl(op_type.c_str())->tiling; | ||
| 2054 | + auto tiling_parse_func = gert::OpImplRegistry::GetInstance().GetOpImpl(op_type.c_str())->tiling_parse; | ||
| 2055 | + | ||
| 2056 | + auto kernel_holder = gert::KernelRunContextFaker() | ||
| 2057 | + .KernelIONum(2, 1) | ||
| 2058 | + .Inputs({const_cast<char*>(compile_info_string.c_str()), | ||
| 2059 | + reinterpret_cast<void*>(&platform_info)}) | ||
| 2060 | + .Outputs({&compile_info}) | ||
| 2061 | + .Build(); | ||
| 2062 | + | ||
| 2063 | + ASSERT_TRUE(kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->Init()); | ||
| 2064 | + kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("SoCInfo", soc_infos); | ||
| 2065 | + kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("AICoreSpec", aicore_spec); | ||
| 2066 | + kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetCoreNumByCoreType("AICore"); | ||
| 2067 | + kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("AICoreintrinsicDtypeMap", | ||
| 2068 | + intrinsics); | ||
| 2069 | + kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("version", soc_version); | ||
| 2070 | + kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("version", npuarchs); | ||
| 2071 | + | ||
| 2072 | + ASSERT_EQ(tiling_parse_func(kernel_holder.GetContext<gert::KernelContext>()), ge::GRAPH_SUCCESS); | ||
| 2073 | + | ||
| 2074 | + auto param = gert::TilingData::CreateCap(4096); | ||
| 2075 | + auto workspace_size_holer = gert::ContinuousVector::Create<size_t>(4096); | ||
| 2076 | + auto ws_size = reinterpret_cast<gert::ContinuousVector*>(workspace_size_holer.get()); | ||
| 2077 | + ASSERT_NE(param, nullptr); | ||
| 2078 | + auto holder = gert::TilingContextFaker() | ||
| 2079 | + .SetOpType(op_type) | ||
| 2080 | + .NodeIoNum(4, 3) | ||
| 2081 | + .IrInstanceNum({1, 1, 1, 1}) | ||
| 2082 | + .InputShapes({&x_shape, &normalized_shape, &gamma_shape, &beta_shape}) | ||
| 2083 | + .OutputShapes({&y_shape, &mean_shape, &rstd_shape}) | ||
| 2084 | + .CompileInfo(&compile_info) | ||
| 2085 | + .PlatformInfo(reinterpret_cast<char*>(&platform_info)) | ||
| 2086 | + .NodeInputTd(0, ge::DT_FLOAT16, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2087 | + .NodeInputTd(1, ge::DT_INT32, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2088 | + .NodeInputTd(2, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2089 | + .NodeInputTd(3, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2090 | + .NodeOutputTd(0, ge::DT_FLOAT16, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2091 | + .NodeOutputTd(1, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2092 | + .NodeOutputTd(2, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND) | ||
| 2093 | + .NodeAttrs({{"epsilon", Ops::NN::AnyValue::CreateFrom<float>(1e-05)}}) | ||
| 2094 | + .TilingData(param.get()) | ||
| 2095 | + .Workspace(ws_size) | ||
| 2096 | + .Build(); | ||
| 2097 | + | ||
| 2098 | + gert::TilingContext* tiling_context = holder.GetContext<gert::TilingContext>(); | ||
| 2099 | + ASSERT_NE(tiling_context->GetPlatformInfo(), nullptr); | ||
| 2100 | + holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("SoCInfo", soc_infos); | ||
| 2101 | + holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("AICoreSpec", aicore_spec); | ||
| 2102 | + holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetCoreNumByCoreType("AICore"); | ||
| 2103 | + holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("AICoreintrinsicDtypeMap", intrinsics); | ||
🟡 Medium Priority 变更行(test_layer_norm_v4_tiling.cpp:2100 与 2130-2133)只在 tiling_parse 上下文设置了 受影响契约: 失败模式: 对照证据:本工程所有 arch35 用例都在 tiling_parse 与 tiling 两个上下文同时设置 修复方向:在 helper 中声明 建议:在 helper 中增加 ![]() ![]() Incorrect? | |||
| 2104 | + holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("version", npuarchs); | ||
| 2105 | + | ||
| 2106 | + ASSERT_EQ(tiling_func(tiling_context), ge::GRAPH_SUCCESS); | ||
| 2107 | + auto tiling_key = tiling_context->GetTilingKey(); | ||
| 2108 | + if (expectedSplits > 0) { | ||
| 2109 | + ASSERT_EQ(tiling_key, 810); | ||
| 2110 | + EXPECT_EQ(tiling_context->GetBlockDim(), a * expectedSplits); | ||
| 2111 | + // SplitR publishes separate 32B mean and variance rows per core. | ||
| 2112 | + // Include the system SyncAll workspace. | ||
| 2113 | + auto platform = platform_ascendc::PlatformAscendC(tiling_context->GetPlatformInfo()); | ||
| 2114 | + auto* workspace = tiling_context->GetWorkspaceSizes(1); | ||
| 2115 | + ASSERT_NE(workspace, nullptr); | ||
| 2116 | + EXPECT_EQ(workspace[0], platform.GetLibApiWorkSpaceSize() + | ||
| 2117 | + 2 * a * expectedSplits * Ops::Base::GetUbBlockSize(tiling_context)); | ||
| 2118 | + } else { | ||
| 2119 | + EXPECT_NE(tiling_key, 810); | ||
| 2120 | + } | ||
| 2121 | + auto tilingData = tiling_context->GetRawTilingData(); | ||
| 2122 | + ASSERT_NE(tilingData, nullptr); | ||
| 2123 | + if (expectedSplits > 0) { | ||
| 2124 | + // The common Welford/SplitR ABI starts with fourteen int64_t fields; | ||
| 2125 | + // read only this prefix, without depending on the epsilon/tail padding. | ||
| 2126 | + int64_t fields[14] = {}; | ||
| 2127 | + ASSERT_GE(tilingData->GetDataSize(), sizeof(fields)); | ||
| 2128 | + std::memcpy(fields, tilingData->GetData(), sizeof(fields)); | ||
| 2129 | + constexpr size_t MAIN_BLOCK_FACTOR_FIELD = 5; | ||
| 2130 | + EXPECT_EQ(fields[MAIN_BLOCK_FACTOR_FIELD], 1); // One row per SplitR core. | ||
| 2131 | + constexpr size_t TILE_LENGTH_FIELD = 7; | ||
| 2132 | + constexpr size_t WELFORD_TEMP_FIELD = 8; | ||
| 2133 | + constexpr size_t API_TEMP_FIELD = 13; | ||
| 2134 | + const int64_t tileLength = fields[TILE_LENGTH_FIELD]; | ||
| 2135 | + ASSERT_GT(tileLength, 0); | ||
| 2136 | + ASSERT_GT(compile_info.vectorLength, 0); | ||
| 2137 | + // FP16 x/y and FP32 gamma/beta, all double buffered; two double | ||
| 2138 | + // buffered statistics queues contain 256 floats each. | ||
| 2139 | + const int64_t ioBuffers = 24 * tileLength + 4 * 256 * sizeof(float); | ||
| 2140 | + const int64_t mergeBuffers = 2 * compile_info.vectorLength; | ||
| 2141 | + EXPECT_LE(ioBuffers + fields[WELFORD_TEMP_FIELD] + fields[API_TEMP_FIELD] + mergeBuffers, | ||
| 2142 | + compile_info.ubSizePlatForm); | ||
| 2143 | + } | ||
| 2144 | +} | ||
| 2145 | + | ||
| 2146 | +TEST_F(LayerNormV4Tiling, split_r_long_row) { CheckLayerNormV4SplitR(1, 1048576, 16); } | ||
| 2147 | + | ||
| 2148 | +TEST_F(LayerNormV4Tiling, split_r_multiple_rows_and_tail) { CheckLayerNormV4SplitR(3, 65537, 4); } | ||
| 2149 | + | ||
| 2150 | +TEST_F(LayerNormV4Tiling, split_r_short_row_fallback) { CheckLayerNormV4SplitR(1, 65535, 0); } | ||
| 2151 | + | ||
| 2152 | +TEST_F(LayerNormV4Tiling, split_r_sufficient_rows_fallback) { CheckLayerNormV4SplitR(64, 65537, 0); } | ||
| 2153 | + | ||
| 2154 | +TEST_F(LayerNormV4Tiling, split_r_threshold_boundary) { CheckLayerNormV4SplitR(1, 65536, 4); } | ||
| 2155 | + | ||
| 2156 | +TEST_F(LayerNormV4Tiling, split_r_max_admitted_rows) { CheckLayerNormV4SplitR(16, 65536, 4); } | ||
| 2157 | + | ||
| 2158 | +TEST_F(LayerNormV4Tiling, split_r_above_max_admitted_rows) { CheckLayerNormV4SplitR(17, 65536, 0); } | ||
| 2159 | + | ||
| 2160 | +TEST_F(LayerNormV4Tiling, split_r_nearly_saturated_rows_fallback) { CheckLayerNormV4SplitR(32, 65536, 0); } | ||
| 2161 | + | ||
| 2162 | +// Non-power-of-two groups exercise compact masked reductions; a tail | ||
| 2163 | +// element also makes the per-core counts unequal in the finalization input. | ||
| 2164 | +TEST_F(LayerNormV4Tiling, split_r_five_parts_with_tail) { CheckLayerNormV4SplitR(1, 81921, 5); } | ||
| 2165 | + | ||
| 2166 | +TEST_F(LayerNormV4Tiling, split_r_six_parts_with_tail) { CheckLayerNormV4SplitR(1, 98305, 6); } | ||
| 2167 | + | ||
| 2168 | +TEST_F(LayerNormV4Tiling, split_r_seven_parts_core_limited) { CheckLayerNormV4SplitR(9, 1048577, 7); } | ||
| 2169 | + | ||
| 2170 | +TEST_F(LayerNormV4Tiling, split_r_nine_parts_with_tail) { CheckLayerNormV4SplitR(1, 147457, 9); } | ||
| 2171 | + | ||
| 2172 | +TEST_F(LayerNormV4Tiling, split_r_fifteen_parts_with_tail) { CheckLayerNormV4SplitR(1, 245761, 15); } | ||
| @@ -94,6 +94,43 @@ inline void InitTilingData(uint8_t* tiling, LayerNormV4TilingDataWelford* const_ | |||
| 94 | } | 94 | } |
| 95 | #endif | 95 | #endif |
| 96 | 96 | ||
| 97 | +#pragma pack(1) | ||
| 98 | +struct LayerNormV4TilingDataSplitR { | ||
| 99 | + int64_t M = 0; | ||
| 100 | + int64_t N = 0; | ||
| 101 | + int64_t rAlign = 0; | ||
| 102 | + int64_t numBlocks = 0; | ||
| 103 | + int64_t mainBlockCount = 0; | ||
| 104 | + int64_t mainBlockFactor = 0; | ||
| 105 | + int64_t tailBlockFactor = 0; | ||
| 106 | + int64_t tileLength = 0; | ||
| 107 | + int64_t welfordTempSize = 0; | ||
| 108 | + int64_t welfordUpdateTimes = 0; | ||
| 109 | + int64_t welfordUpdateTail = 0; | ||
| 110 | + int64_t nullptrGamma = 0; | ||
| 111 | + int64_t nullptrBeta = 0; | ||
| 112 | + int64_t apiTempBufferSize = 0; | ||
| 113 | + float epsilon = 0; | ||
| 114 | + int32_t pad_ = 0; // Match the host tiling serialization alignment before splitR. | ||
| 115 | + int64_t splitR = 0; | ||
| 116 | +}; | ||
| 117 | +#pragma pack() | ||
| 118 | + | ||
| 119 | +#ifdef __NPU_TILING__ | ||
| 120 | +inline[aicore] void InitTilingData(const __gm__ uint8_t* tiling, LayerNormV4TilingDataSplitR* const_data) | ||
| 121 | +{ | ||
| 122 | + const __gm__ uint32_t* src = (const __gm__ uint32_t*)tiling; | ||
| 123 | + uint32_t* dst = (uint32_t*)const_data; | ||
| 124 | + for (auto i = 0; i < sizeof(LayerNormV4TilingDataSplitR) / 4; i++) | ||
| 125 | + *(dst + i) = *(src + i); | ||
| 126 | +} | ||
| 127 | +#else | ||
| 128 | +inline void InitTilingData(uint8_t* tiling, LayerNormV4TilingDataSplitR* const_data) | ||
| 129 | +{ | ||
| 130 | + memcpy(const_data, tiling, sizeof(LayerNormV4TilingDataSplitR)); | ||
| 131 | +} | ||
| 132 | +#endif | ||
| 133 | + | ||
| 97 | #define GET_TILING_DATA_WITH_STRUCT(tiling_struct, tiling_data, tiling_arg) \ | 134 | #define GET_TILING_DATA_WITH_STRUCT(tiling_struct, tiling_data, tiling_arg) \ |
| 98 | tiling_struct tiling_data; \ | 135 | tiling_struct tiling_data; \ |
| 99 | InitTilingData(tiling_arg, &tiling_data) | 136 | InitTilingData(tiling_arg, &tiling_data) |


[Minor] DoLibApiTiling 恢复 ubSizePlatForm 后,需确认后续阶段不会再按全量 UB 规划缓冲
这里在基类
DoLibApiTiling()前后对ubSizePlatForm做「扣减 partial 队列 → 计算 tileLength → 恢复」。前提是:基类DoOpTiling()及后续阶段不再读取ubSizePlatForm规划 UB 缓冲,否则partialIn_/partialOut_((S+1)×32B)会与基类 welfordTemp/apiTemp/输入输出队列叠加,最终超出 UB。请确认:
LayerNormV4WelfordTiling::DoOpTiling()以及 SplitR 之后的所有 UB 分配都只依赖tileLength(而非完整ubSizePlatForm);reserved=(splitR_+1)*32与 kernel 实际partialOut_(32B) + partialIn_(S×32B)一致(当前一致)。