Merged
perf(layer_norm_v4): 独立SplitR执行类实现小A大R跨核归约 #13181
perf(layer_norm_v4): 独立SplitR执行类实现小A大R跨核归约 #13181
Merged
pipimanAdd on 23 days ago
Total: 11 files changed+1174-416
@@ -84,6 +84,7 @@ CANN算子开发者需要同时维护Host端和Kernel端代码。
84| 400-422 | Parallel Welford | 基数400 = Welford算法族的起始编号 |84| 400-422 | Parallel Welford | 基数400 = Welford算法族的起始编号 |
85| 300-322 | Two-Pass + Binary累加 | 基数300 = TwoPass算法族的起始编号 |85| 300-322 | Two-Pass + Binary累加 | 基数300 = TwoPass算法族的起始编号 |
86| 500-522 | Two-Pass Perf(优化版) | 基数500 = TwoPassPerf算法族的起始编号 |86| 500-522 | Two-Pass Perf(优化版) | 基数500 = TwoPassPerf算法族的起始编号 |
87+| 800-822 | SplitR 跨核分段归约 | 基数800 = SplitR算法族的起始编号 |
87 88 
88每个算法族内的取值范围(如400-422)覆盖了不同的**数据类型组合**(如`FLOAT_FLOAT`、`HALF_FLOAT`、`BF16_FLOAT`等),即每个具体的`(src_dtype, accum_dtype)`组合占用一个TILING_KEY值。因此`400-422`意味着Welford族覆盖了约23种数据类型组合。这些范围值的设定是CANN框架的命名约定,与底层实现无直接数学关联。89每个算法族内的取值范围(如400-422)覆盖了不同的**数据类型组合**(如`FLOAT_FLOAT`、`HALF_FLOAT`、`BF16_FLOAT`等),即每个具体的`(src_dtype, accum_dtype)`组合占用一个TILING_KEY值。因此`400-422`意味着Welford族覆盖了约23种数据类型组合。这些范围值的设定是CANN框架的命名约定,与底层实现无直接数学关联。
89 90 
@@ -98,7 +99,7 @@ if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) { ... }
98 99 
99**TILING_KEY与Tiling模板优先级的关系:**100**TILING_KEY与Tiling模板优先级的关系:**
100 101 
101-Tiling模板注册时附带一个优先级数值(如Welford为4000,TwoPass为200,TwoPassPerf为150)。框架按优先级从高到低尝试匹配:先检查Welford(4000)是否满足`IsCapable()`,不满足则尝试TwoPass(200),再不满足则尝试TwoPassPerf(150)。这与TILING_KEY的范围值(400/300/500)是两套独立的编号体系 —— 前者控制模板匹配顺序,后者是算法标识符。102+Tiling模板注册时附带一个优先级数值(如SplitR为140,TwoPassPerf为150,TwoPass为200,Welford为4000)。框架按优先级数值从小到大依次尝试匹配:先检查SplitR(140)是否满足`IsCapable()`,不满足则尝试TwoPassPerf(150),再不满足则尝试TwoPass(200),最后尝试Welford(4000)。这与TILING_KEY的范围值(400/300/500/800)是两套独立的编号体系 —— 前者控制模板匹配顺序,后者是算法标识符。
102 103 
103#### 0.3.4 regbase(寄存器基址寻址模式)104#### 0.3.4 regbase(寄存器基址寻址模式)
104 105 
@@ -108,7 +109,7 @@ Tiling模板注册时附带一个优先级数值(如Welford为4000,TwoPass
108- **支持向量融合**:部分CANN向量融合指令(如`FusedMulDstAdd`)要求在regbase模式下才能使用109- **支持向量融合**:部分CANN向量融合指令(如`FusedMulDstAdd`)要求在regbase模式下才能使用
109- **减少UB访问冲突**:通过寄存器中转减少了UB的读取次数110- **减少UB访问冲突**:通过寄存器中转减少了UB的读取次数
110 111 
111-`isRegBase`是一个由Host端根据输入shape和数据类型计算出的布尔值,通过tiling参数传递给Kernel端。如果当前shape不满足regbase的硬件要求(如对齐约束),`isRegBase`会被置为`false`,此时三个算法分支均不会被选中,算子回退到通用(非regbase)路径。112+`isRegBase`是一个由Host端根据输入shape和数据类型计算出的布尔值,通过tiling参数传递给Kernel端。如果当前shape不满足regbase的硬件要求(如对齐约束),`isRegBase`会被置为`false`,此时四个算法分支均不会被选中,算子回退到通用(非regbase)路径。
112 113 
113**为什么Welford和TwoPass都要求isRegBase == true?**因为并行Welford和二分累加算法需要频繁访问M2累加器、临时缓冲区等,regbase模式能保证这些访问以最高效的方式进行。当前不存在不需要regbase的Welford/TwoPass降级路径 —— 如果不满足regbase条件,算子会回退到其他LayerNorm实现(如V3或通用路径)。114**为什么Welford和TwoPass都要求isRegBase == true?**因为并行Welford和二分累加算法需要频繁访问M2累加器、临时缓冲区等,regbase模式能保证这些访问以最高效的方式进行。当前不存在不需要regbase的Welford/TwoPass降级路径 —— 如果不满足regbase条件,算子会回退到其他LayerNorm实现(如V3或通用路径)。
114 115 
@@ -195,19 +196,22 @@ y = gamma * ((x - mean) * rstd) + beta // 归一化 + 仿射变换
195 196 
196## 2. 总体架构197## 2. 总体架构
197 198 
198-LayerNormV4在950(arch35)平台上有三个算法分支,通过**TILING_KEY**在编译期动态选择:199+LayerNormV4在950(arch35)平台上有四个算法分支,通过**TILING_KEY**在编译期动态选择:
199 200 
200| TILING_KEY | 算法 | 模板优先级 | 适用场景 |201| TILING_KEY | 算法 | 模板优先级 | 适用场景 |
201|-----------|------|--------|---------|202|-----------|------|--------|---------|
202-| 400-422 | Parallel Welford | 最高(4000) | 通用场景,单遍扫描 |203+| 800-822 | SplitR 跨核分段归约 | 最高(140,最先尝试) | 小A大R:R≥65536 且 A≤AIV核数/4,每核扫描连续片段后经workspace交换、一次全核同步与固定顺序合并 |
204+| 400-422 | Parallel Welford | 最低(4000,最后尝试) | 通用场景,单遍扫描 |
203| 300-322 | Two-Pass + Binary累加 | 中(200) | 超长轴场景(R较大,Welford不满足UB约束时) |205| 300-322 | Two-Pass + Binary累加 | 中(200) | 超长轴场景(R较大,Welford不满足UB约束时) |
204-| 500-522 | Two-Pass Perf | 低(150) | 中等长度轴,极致性能(额外要求rowAlign不超硬限制、colSize/rowAlign不触发联立阈值) |206+| 500-522 | Two-Pass Perf | 次之(150) | 中等长度轴,极致性能(额外要求rowAlign不超硬限制、colSize/rowAlign不触发联立阈值) |
205 207 
206-**分派逻辑**(入口代码位于`op_kernel/layer_norm_v4_apt.h:80-143`):208+**分派逻辑**(入口代码位于`op_kernel/layer_norm_v4_apt.h:104-184`):
207 209 
208```cpp210```cpp
209__aicore__ inline void layer_norm_impl(...) {211__aicore__ inline void layer_norm_impl(...) {
210- if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) {212+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT)) {
213+ RegbaseSplitRImpl<float, float>(...);
214+ } else if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) {
211 RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...);215 RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...);
212 } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) {216 } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) {
213 RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...);217 RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...);
@@ -217,7 +221,7 @@ __aicore__ inline void layer_norm_impl(...) {
217}221}
218```222```
219 223 
220-`TILING_KEY_IS()`宏在编译期展开为常量比较,三个分支中只有匹配的一个会被保留,其余在编译优化阶段被删除,无运行时开销。宏的参数命名规则参见 [前置知识0.3.3](#033-tiling_key--算法选择键)。224+`TILING_KEY_IS()`宏在编译期展开为常量比较,四个分支中只有匹配的一个会被保留,其余在编译优化阶段被删除,无运行时开销。宏的参数命名规则参见 [前置知识0.3.3](#033-tiling_key--算法选择键)。
221 225 
222---226---
223 227 
@@ -409,7 +413,7 @@ Level 2: (s0..s3) (s4..s7) ← 仍相近
409Level 3: 最终总和(s0..s7)413Level 3: 最终总和(s0..s7)
410```414```
411 415 
412-**TwoPass的空间与性能代价:**两遍扫描意味着数据从Global Memory搬入UB至少3次(第1遍算均值、第2遍算方差、归一化再1次),相比Welford的2次多了约50% 的带宽开销。这是TwoPass的优先级低于Welford的核心理由。但它的优势在于**中间状态极紧凑** —— 仅需存放二分累加树的临时缓冲区,不依赖每行的M2累加器。416+**TwoPass的空间与性能代价:**两遍扫描意味着数据从Global Memory搬入UB至少3次(第1遍算均值、第2遍算方差、归一化再1次),相比Welford的2次多了约50% 的带宽开销。额外的带宽开销是模板准入与选择时权衡的因素之一。但它的优势在于**中间状态极紧凑** —— 仅需存放二分累加树的临时缓冲区,不依赖每行的M2累加器。
413 417 
414---418---
415 419 
@@ -522,23 +526,24 @@ CANN框架在Host端按照注册的优先级顺序依次尝试匹配Tiling模板
522 526 
523| 优先级 | 模板 | 文件 |527| 优先级 | 模板 | 文件 |
524|-------|------|------|528|-------|------|------|
529+| 140 | LayerNormV4SplitRTiling | `op_host/layer_norm_v4_split_r_tiling.cpp` |
525| 4000 | LayerNormV4WelfordTiling | `op_host/layer_norm_v4_welford_tiling.cpp` |530| 4000 | LayerNormV4WelfordTiling | `op_host/layer_norm_v4_welford_tiling.cpp` |
526| 200 | LayerNormV4RegBaseTwoPassTiling | `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` |531| 200 | LayerNormV4RegBaseTwoPassTiling | `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` |
527| 150 | LayerNormV4RegBaseTwoPassPerfTiling | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` |532| 150 | LayerNormV4RegBaseTwoPassPerfTiling | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` |
528 533 
529-优先级4000/200/150是模板**匹配优先级**(越高越先尝试),与TILING_KEY的范围值(400/300/500)是两套独立编号。框架先按优先级找模板,模板再打出对应的TILING_KEY。534+模板按优先级数值从小到大尝试:SplitR(140)→TwoPassPerf(150)→TwoPass(200)→Welford(4000)。模板优先级决定匹配顺序,TILING_KEY 标识最终选中的算法与类型组合。
530 535 
531-**选择逻辑:**536+SplitR 使用公共 `isRegBase` 平台判断,不额外限制确定性等级。准入形状为 `1 ≤ A ≤ floor(AIV核数/4)`、`65536 ≤ R ≤ INT32_MAX`;64个AIV核时即 `A ≤ 16`。每行分配 `min(16, floor(AIV核数/A), floor(R/16384))` 个核,至少2个核。其他形状继续按优先级尝试原模板。
532 537 
533-1. **Welford优先(4000)**:当`isRegBase == true`且R不超过UB容量限制时被选中。Welford是首选的默认算法 —— 单遍扫描,精度最优,性价比最高538+每核以 Welford 计算连续片段的均值和总体方差,写入独立的 mean、variance workspace 平面;`SyncAll()` 后通过 `DataCopyPad<float, PaddingMode::Compact>` 将每核记录中的首个 float 紧凑搬入 UB。两个平面各占一个向量,空间系数2表示两个统计量,并非双缓冲。最多16段的统计量使用掩码向量归约,不再需要通用 BatchNorm 的多级归约暂存。
534-2. **TwoPass备选(200)**:在Welford不可用时生效。Welford不可用的典型场景是R过大导致M2累加器数组超出UB容量。此时TwoPass的紧凑中间状态使其成为唯一可行的regbase选项539+ 
535-3. **TwoPassPerf降级(150)**:优先级最低。在其适用范围内(中等轴长+满足附加约束),通过(x-mean) 复用和行配对优化,性能在适用shape下可达到更优540+令 `w_s = n_s / R`,以首段均值为参考计算 `mean = mean_0 + sum((mean_s - mean_0) * w_s)`,再计算 `variance = sum(variance_s * w_s + (mean_s - mean)^2 * w_s)`。权重先归一化再乘方差,避免 `variance_s * n_s` 的中间值溢出;平方项仍受 FP32 有限范围限制。每核归一化自身片段,仅首段输出 mean/rstd。
536 541 
537---542---
538 543 
539### 5.2 IsCapable() 检查条件详解544### 5.2 IsCapable() 检查条件详解
540 545 
541-每个模板都有一个`IsCapable()`方法,在Host端运行时根据实际shape判断该模板是否可用。下表完整解析三个算法的Capable条件及其设计意图:546+每个模板都有一个`IsCapable()`方法,在Host端运行时根据实际shape判断该模板是否可用。下表列出原有三个算法的Capable条件;SplitR准入见5.1:
542 547 
543| 算法 | IsCapable条件 | 条件解释与设计意图 |548| 算法 | IsCapable条件 | 条件解释与设计意图 |
544|------|---------------|------------------|549|------|---------------|------------------|
@@ -548,9 +553,9 @@ CANN框架在Host端按照注册的优先级顺序依次尝试匹配Tiling模板
548 553 
549**TwoPassPerf的附加条件解释:**554**TwoPassPerf的附加条件解释:**
550 555 
551-- **rowAlign不超出硬限制**:`IsCapable()`的第一条几何判断(约第63行)为`rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`时返回false,即约`4 × VL_F32²`这一量级(`LN_NUM_TWO = 2`,代入后为`2 × 64 × 64 × 2 = 16384`)。这意味着rowAlign存在一个与`VL_F32²`相关的硬上限,超出则不可用。556+- **rowAlign不超出硬限制**:`IsCapable()`的第一条几何判断(`op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp:70-73`)为`rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`时返回false,即约`4 × VL_F32²`这一量级(`LN_NUM_TWO = 2`,代入后为`2 × 64 × 64 × 2 = 16384`)。这意味着rowAlign存在一个与`VL_F32²`相关的硬上限,超出则不可用。
552 557 
553-- **colSize/rowAlign不触发联立阈值**:第二条判断(约第68行)为`colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`时返回false。其中两个阈值为静态常量(= 4096和8192),分别对应colSize和rowAlign的上限。两条件需同时满足才触发拒绝 —— 这避免了单独一个维度偏大但另一个维度较小时误判不可用。558+- **colSize/rowAlign不触发联立阈值**:第二条判断(第76-78行)为`colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`时返回false。其中两个阈值为静态常量(= 4096和8192),分别对应colSize和rowAlign的上限。两条件需同时满足才触发拒绝 —— 这避免了单独一个维度偏大但另一个维度较小时误判不可用。
554 559 
555- **满足UB容量约束**:通过`CanFitInBuffer(1)`校验tile大小 + (x-mean) 差值缓冲区 + 其他临时空间总和不超过UB容量。560- **满足UB容量约束**:通过`CanFitInBuffer(1)`校验tile大小 + (x-mean) 差值缓冲区 + 其他临时空间总和不超过UB容量。
556 561 
@@ -562,7 +567,7 @@ TwoPass的IsCapable条件: isRegBase + UB约束
562TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB约束 ← 最严格567TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB约束 ← 最严格
563```568```
564 569 
565-框架从Welford(最宽松)开始检查,若满足则直接选中;不满足才逐级尝试更严格的分支。这与模板优先级(4000→200→150)的设计完全一致。570+以上条件宽松程度不代表匹配顺序。框架按 SplitR→TwoPassPerf→TwoPass→Welford 的顺序检查,首个成功模板负责生成 tiling。
566 571 
567### 5.3 三维权衡分析:精度、性能与适用性572### 5.3 三维权衡分析:精度、性能与适用性
568 573 
@@ -574,7 +579,7 @@ TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB
574| **TwoPass** | 超长轴适应性 | 多一遍全局数据搬运(3次vs 2次) | R小时性能损失明显,无精度优势 |579| **TwoPass** | 超长轴适应性 | 多一遍全局数据搬运(3次vs 2次) | R小时性能损失明显,无精度优势 |
575| **TwoPassPerf** | 极致性能 + 极好精度 | rowAlign和colSize/rowAlign受联立阈值限制 | shape越界时不可用,回退到较慢分支 |580| **TwoPassPerf** | 极致性能 + 极好精度 | rowAlign和colSize/rowAlign受联立阈值限制 | shape越界时不可用,回退到较慢分支 |
576 581 
577-**设计哲学:**三套算法不是互相替代的关系,而是对"**shape空间**"的一个覆盖性划分。框架通过优先级(4000→200→150)对常见case做最优配置,通过IsCapable条件在极端case下做兜底保证。582+**设计哲学:**三套算法不是互相替代的关系,而是对"**shape空间**"的一个覆盖性划分。框架通过优先级(140→150→200→4000)对常见case做最优配置,通过IsCapable条件在极端case下做兜底保证。
578 583 
579对于软件领域的读者来说,这种"多算法覆盖shape空间"的模式类似于:584对于软件领域的读者来说,这种"多算法覆盖shape空间"的模式类似于:
580 585 
@@ -596,7 +601,7 @@ TwoPassPerf的IsCapable条件: isRegBase + rowAlign硬限制 + 联立阈值 + UB
596| `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Host CPU | TwoPass Perf tiling(含IsCapable中的A/R阈值定义) |601| `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Host CPU | TwoPass Perf tiling(含IsCapable中的A/R阈值定义) |
597| `op_kernel/layer_norm_v4_apt.h` | AI Core(设备端) | 950平台kernel入口和TILING_KEY分派 |602| `op_kernel/layer_norm_v4_apt.h` | AI Core(设备端) | 950平台kernel入口和TILING_KEY分派 |
598| `op_host/layer_norm_v4_tiling.h` | Host CPU | Tiling数据结构定义(各模板共享) |603| `op_host/layer_norm_v4_tiling.h` | Host CPU | Tiling数据结构定义(各模板共享) |
599-| `op_host/layer_norm_v4_def.cpp:191` | Host CPU | 950平台配置,注册为`layer_norm_v4_apt` |604+| `op_host/layer_norm_v4_def.cpp:171` | Host CPU | 950平台配置,注册为`layer_norm_v4_apt` |
600 605 
601代码分层小结:606代码分层小结:
602 607 
@@ -612,7 +617,7 @@ LayerNormV4在Ascend 950平台上的精度优化体现了两个层面的设计
612 617 
613- **算法层面**:Parallel Welford通过在线递推更新方式将中心化操作融入每一步,从根源上避免了朴素方差公式中`E[X²] - (E[X])²`大数相减的灾难性抵消(catastrophic cancellation);超长轴二分累加通过树状归约替代链式加法,使每一步相加的两个操作数量级相近,将累加舍入误差从O(n) 降至O(log n)618- **算法层面**:Parallel Welford通过在线递推更新方式将中心化操作融入每一步,从根源上避免了朴素方差公式中`E[X²] - (E[X])²`大数相减的灾难性抵消(catastrophic cancellation);超长轴二分累加通过树状归约替代链式加法,使每一步相加的两个操作数量级相近,将累加舍入误差从O(n) 降至O(log n)
614 619 
615-- **工程层面**:三套算法通过TILING_KEY机制在编译期动态选择,以模板优先级(4000/200/150)和IsCapable条件构成两级筛选,不同算法分别追求精度(Welford)、极端适应性(TwoPass)、极致性能(TwoPassPerf),共同覆盖了从短轴到超长轴的完整shape空间;UB空间的精细化管理(double buffer、tile大小对齐、临时缓冲区容量校验)确保了数据搬运与计算在AI Core上的高效流水620+- **工程层面**:三套算法通过TILING_KEY机制在编译期动态选择,以模板优先级(140/150/200/4000)和IsCapable条件构成两级筛选,不同算法分别追求精度(Welford)、极端适应性(TwoPass)、极致性能(TwoPassPerf),共同覆盖了从短轴到超长轴的完整shape空间;UB空间的精细化管理(double buffer、tile大小对齐、临时缓冲区容量校验)确保了数据搬运与计算在AI Core上的高效流水
616 621 
617这两种算法思想不仅适用于LayerNormV4,对于所有需要沿超长轴做归约的Norm类算子(如 **RMSNorm**、**GroupNorm**、**InstanceNorm**等),以及更广义的高精度浮点归约场景,都具有普适的参考价值。622这两种算法思想不仅适用于LayerNormV4,对于所有需要沿超长轴做归约的Norm类算子(如 **RMSNorm**、**GroupNorm**、**InstanceNorm**等),以及更广义的高精度浮点归约场景,都具有普适的参考价值。
618 623 
@@ -81,6 +81,7 @@ Similarly, `LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT` indicates the Two-Pass (binary ac
81 81 
82| TILING_KEY range | Algorithm | Meaning of the base |82| TILING_KEY range | Algorithm | Meaning of the base |
83|----------------|------|---------|83|----------------|------|---------|
84+| 800-822 | SplitR | Base 800 = starting number of the SplitR algorithm family |
84| 400-422 | Parallel Welford | Base 400 = starting number of the Welford algorithm family |85| 400-422 | Parallel Welford | Base 400 = starting number of the Welford algorithm family |
85| 300-322 | Two-Pass + Binary accumulation | Base 300 = starting number of the TwoPass algorithm family |86| 300-322 | Two-Pass + Binary accumulation | Base 300 = starting number of the TwoPass algorithm family |
86| 500-522 | Two-Pass Perf (optimized version) | Base 500 = starting number of the TwoPassPerf algorithm family |87| 500-522 | Two-Pass Perf (optimized version) | Base 500 = starting number of the TwoPassPerf algorithm family |
@@ -98,7 +99,7 @@ This lets the compiler eliminate mismatched branches at compile time and remove
98 99 
99**Relationship between TILING_KEY and Tiling template priority:**100**Relationship between TILING_KEY and Tiling template priority:**
100 101 
101-When a Tiling template is registered, it carries a priority value (for example, Welford is 4000, TwoPass is 200, and TwoPassPerf is 150). The framework tries matching from the highest priority to the lowest: it first checks whether Welford (4000) satisfies `IsCapable()`; if not, it tries TwoPass (200); if that still fails, it tries TwoPassPerf (150). This priority system and the TILING_KEY range values (400/300/500) are two independent numbering systems — the former controls the template matching order, and the latter is the algorithm identifier.102+When a Tiling template is registered, it carries a priority value (for example, SplitR is 140, TwoPassPerf is 150, TwoPass is 200, and Welford is 4000). The framework tries matching in ascending priority order: it first checks whether SplitR (140) satisfies `IsCapable()`; if not, it tries TwoPassPerf (150), then TwoPass (200), and finally Welford (4000). This priority system and the TILING_KEY range values (400/300/500/800) are two independent numbering systems — the former controls the template matching order, and the latter is the algorithm identifier.
102 103 
103#### 0.3.4 regbase (Register Base Addressing Mode)104#### 0.3.4 regbase (Register Base Addressing Mode)
104 105 
@@ -193,21 +194,24 @@ On the Ascend 950 platform, the LayerNormV4 operator uses two core algorithms, *
193 194 
194---195---
195 196 
196-## 2. Overall Architecture197+## 2. Architecture
197 198 
198-LayerNormV4 has three algorithm branches on the 950 (arch35) platform, dynamically selected at compile time through **TILING_KEY**:199+LayerNormV4 has four algorithm branches on the 950 (arch35) platform, dynamically selected at compile time through **TILING_KEY**:
199 200 
200| TILING_KEY | Algorithm | Template Priority | Applicable Scenario |201| TILING_KEY | Algorithm | Template Priority | Applicable Scenario |
201|-----------|------|--------|---------|202|-----------|------|--------|---------|
202-| 400-422 | Parallel Welford | Highest (4000) | General scenarios, single-pass scan |203+| 800-822 | SplitR cross-core segmented reduction | Highest (140, tried first) | Small A, large R: R≥65536 and A≤AIV cores/4; each core scans a contiguous segment, then exchanges via workspace, one whole-core synchronization, and a fixed-order merge |
204+| 400-422 | Parallel Welford | Lowest (4000, tried last) | General scenarios, single-pass scan |
203| 300-322 | Two-Pass + Binary accumulation | Medium (200) | Extra-long axis scenarios (R is large and Welford does not satisfy the UB constraint) |205| 300-322 | Two-Pass + Binary accumulation | Medium (200) | Extra-long axis scenarios (R is large and Welford does not satisfy the UB constraint) |
204-| 500-522 | Two-Pass Perf | Low (150) | Medium-length axes, extreme performance (additional requirements: rowAlign does not exceed the hard limit, and colSize/rowAlign does not trigger the joint threshold) |206+| 500-522 | Two-Pass Perf | Next (150) | Medium-length axes, extreme performance (additional requirements: rowAlign does not exceed the hard limit, and colSize/rowAlign does not trigger the joint threshold) |
205 207 
206-**Dispatch logic** (entry code in `op_kernel/layer_norm_v4_apt.h:80-143`):208+**Dispatch logic** (entry code in `op_kernel/layer_norm_v4_apt.h:104-184`):
207 209 
208```cpp210```cpp
209__aicore__ inline void layer_norm_impl(...) {211__aicore__ inline void layer_norm_impl(...) {
210- if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) {212+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT)) {
213+ RegbaseSplitRImpl<float, float>(...);
214+ } else if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) {
211 RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...);215 RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(...);
212 } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) {216 } else if (TILING_KEY_IS(LNV4_REGBASE_WELFORD_FLOAT_FLOAT)) {
213 RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...);217 RegbaseWelfordImpl<float, float, DTYPE_MEAN>(...);
@@ -217,9 +221,9 @@ __aicore__ inline void layer_norm_impl(...) {
217}221}
218```222```
219 223 
220-The `TILING_KEY_IS()` macro expands into a constant comparison at compile time. Only the matching one of the three branches is retained, and the others are removed during compile-time optimization, with no runtime overhead. For the naming rules of macro parameters, see [Prerequisite Knowledge 0.3.3](#033-tiling_key--algorithm-selection-key).224+The `TILING_KEY_IS()` macro expands into a constant comparison at compile time. Only the matching one of the four branches is retained, and the others are removed during compile-time optimization, with no runtime overhead. For the naming rules of macro parameters, see [Prerequisite Knowledge 0.3.3](#033-tiling_key--algorithm-selection-key).
221 225 
222----226+Welford and SplitR share buffer initialization, local statistics, normalization, and cache management in `LayerNormV4WelfordBase`. The whole-row Welford class controls row iteration. SplitR distributes one row across multiple cores and merges segment statistics. Unmatched SplitR shapes continue through Two-Pass Perf, Two-Pass, and Welford template selection. The selected key also encodes the supported input and parameter data types.
223 227 
224## 3. Parallel Welford Algorithm228## 3. Parallel Welford Algorithm
225 229 
@@ -409,7 +413,7 @@ Level 2: (s0..s3) (s4..s7) ← still similar
409Level 3: final sum (s0..s7)413Level 3: final sum (s0..s7)
410```414```
411 415 
412-**Space and performance cost of TwoPass:** Two passes mean the data is moved from Global Memory into UB at least 3 times (the first pass computes the mean, the second pass computes the variance, and normalization takes 1 more time), about 50% more bandwidth overhead than the 2 times of Welford. This is the core reason why TwoPass has a lower priority than Welford. But its advantage is an **extremely compact intermediate state** — it only needs to store the temporary buffer of the binary accumulation tree and does not depend on per-row M2 accumulators.416+**Space and performance cost of TwoPass:** Two passes mean the data is moved from Global Memory into UB at least 3 times (the first pass computes the mean, the second pass computes the variance, and normalization takes 1 more time), about 50% more bandwidth overhead than the 2 times of Welford. Its additional bandwidth cost is one tradeoff considered by template admission and selection. But its advantage is an **extremely compact intermediate state** — it only needs to store the temporary buffer of the binary accumulation tree and does not depend on per-row M2 accumulators.
413 417 
414---418---
415 419 
@@ -514,31 +518,45 @@ The `LAST_LOOP_NUMS` template parameter controls the merge strategy of the **las
514 518 
515---519---
516 520 
517-## 5. Algorithm Selection Strategy521+## 5. Algorithm Selection
518 522 
519-### 5.1 Tiling Template Registration Priorities523+### 5.1 Priority and SplitR admission
520 524 
521-On the Host side, the CANN framework tries to match Tiling templates one by one in the registered priority order:525+Templates are attempted in this order:
522 526 
523-| Priority | Template | File |527+1. SplitR: priority 140.
524-|-------|------|------|528+2. Two-Pass Perf: priority 150.
525-| 4000 | LayerNormV4WelfordTiling | `op_host/layer_norm_v4_welford_tiling.cpp` |529+3. Two-Pass: priority 200.
526-| 200 | LayerNormV4RegBaseTwoPassTiling | `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` |530+4. Welford: priority 4000.
527-| 150 | LayerNormV4RegBaseTwoPassPerfTiling | `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` |
528 531 
529-The priorities 4000/200/150 are the **matching priorities** of templates (higher values are tried first) and are independent of the TILING_KEY range values (400/300/500). The framework first finds a template by priority, and the template then produces the corresponding TILING_KEY.532+SplitR uses the shared `isRegBase` platform check and excludes the V3 path. It does not add a deterministic-level restriction. Its shape conditions are:
530 533 
531-**Selection logic:**534+```text
535+1 <= A <= floor(AIV_core_count / 4)
536+65536 <= R <= INT32_MAX
537+splitR = min(16, floor(AIV_core_count / A), floor(R / 16384))
538+splitR >= 2
539+```
532 540 
533-1. **Welford first (4000)**: selected when `isRegBase == true` and R does not exceed the UB capacity limit. Welford is the preferred default algorithm — single-pass scan, best precision, and best cost-effectiveness541+For 64 AIV cores, A must be at most 16. Segment boundaries are aligned, and the last segment uses its actual element count.
534-2. **TwoPass as fallback (200)**: takes effect when Welford is unavailable. The typical scenario where Welford is unavailable is that R is so large that the M2 accumulator array exceeds the UB capacity. At this point, the compact intermediate state of TwoPass makes it the only feasible regbase option
535-3. **TwoPassPerf downgrade (150)**: lowest priority. Within its applicable range (medium axis length + additional constraints satisfied), through (x-mean) reuse and row pairing optimizations, its performance can be better for the applicable shapes
536 542 
537----543+Each core computes a segment mean and population variance and publishes them into separate workspace planes. Each GM record occupies one UB block. After `SyncAll`, `DataCopyPad<float, PaddingMode::Compact>` gathers the first float of each record into adjacent UB lanes. Each of the two planes is rounded up to a whole number of vectors; the factor two represents mean and variance, independently of double buffering.
544+ 
545+At most sixteen segment statistics fit in one vector. The merge builds weighted contributions in vector-sized chunks and invokes the common `NormCommon::NormCommonRegbase::CalculateReduceSum` interface:
546+ 
547+```text
548+weight_s = count_s / R
549+mean = mean_0 + sum((mean_s - mean_0) * weight_s)
550+variance = sum(variance_s * weight_s + (mean_s - mean)^2 * weight_s)
551+```
552+ 
553+Centering the mean around the first segment retains the large-offset precision fix. Normalizing counts before multiplying variance avoids the intermediate `variance_s * count_s` overflow. The squared mean-difference term remains subject to FP32 range limits. No general multi-level BatchNorm reduction scratch is required.
554+ 
555+Each core normalizes its own input segment. Only the first segment writes mean/rstd. Workspace includes the system synchronization allocation plus the two statistic planes. Host UB budgeting reserves both vector-aligned compact planes before selecting tile length.
538 556 
539### 5.2 Detailed IsCapable() Check Conditions557### 5.2 Detailed IsCapable() Check Conditions
540 558 
541-Each template has an `IsCapable()` method that determines at runtime on the Host side whether the template is available for the actual shape. The following table fully analyzes the Capable conditions of the three algorithms and their design intent:559+Each template has an `IsCapable()` method that determines at runtime on the Host side whether the template is available for the actual shape. The following table lists the Capable conditions of the original three algorithms; SplitR admission is described in 5.1:
542 560 
543| Algorithm | IsCapable Condition | Condition Explanation and Design Intent |561| Algorithm | IsCapable Condition | Condition Explanation and Design Intent |
544|------|---------------|------------------|562|------|---------------|------------------|
@@ -548,21 +566,20 @@ Each template has an `IsCapable()` method that determines at runtime on the Host
548 566 
549**Explanation of the additional conditions of TwoPassPerf:**567**Explanation of the additional conditions of TwoPassPerf:**
550 568 
551-- **rowAlign does not exceed the hard limit**: the first geometric judgment of `IsCapable()` (about line 63) returns false when `rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`, that is, the magnitude of about `4 × VL_F32²` (`LN_NUM_TWO = 2`, substituting gives `2 × 64 × 64 × 2 = 16384`). This means rowAlign has a hard upper limit related to `VL_F32²`; beyond it, the template is unavailable.569+- **rowAlign does not exceed the hard limit**: the first geometric judgment of `IsCapable()` (`op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp:70-73`) returns false when `rowAlign > LN_NUM_TWO * vlFp32 * vlFp32 * LN_NUM_TWO`, that is, the magnitude of about `4 × VL_F32²` (`LN_NUM_TWO = 2`, substituting gives `2 × 64 × 64 × 2 = 16384`). This means rowAlign has a hard upper limit related to `VL_F32²`; beyond it, the template is unavailable.
552 570 
553-- **colSize/rowAlign does not trigger the joint threshold**: the second judgment (about line 68) returns false when `colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`. The two thresholds are static constants (= 4096 and 8192), corresponding to the upper limits of colSize and rowAlign respectively. Both conditions must be satisfied simultaneously to trigger rejection — this avoids misjudging unavailability when only one dimension is too large while the other is small.571+- **colSize/rowAlign does not trigger the joint threshold**: the second judgment (lines 76-78) returns false when `colSize >= LN_ROW_THRESHOLD && rowAlign >= LN_COL_THRESHOLD`. The two thresholds are static constants (= 4096 and 8192), corresponding to the upper limits of colSize and rowAlign respectively. Both conditions must be satisfied simultaneously to trigger rejection — this avoids misjudging unavailability when only one dimension is too large while the other is small.
554 572 
555- **UB capacity constraint satisfied**: `CanFitInBuffer(1)` validates that the total of the tile size + the (x-mean) difference buffer + other temporary space does not exceed the UB capacity.573- **UB capacity constraint satisfied**: `CanFitInBuffer(1)` validates that the total of the tile size + the (x-mean) difference buffer + other temporary space does not exceed the UB capacity.
556 574 
557**Logical relationship between the conditions (from strict to relaxed):**575**Logical relationship between the conditions (from strict to relaxed):**
558- 
559```Cpp576```Cpp
560Welford IsCapable condition: isRegBase == true ← most relaxed577Welford IsCapable condition: isRegBase == true ← most relaxed
561TwoPass IsCapable condition: isRegBase + UB constraint ← medium578TwoPass IsCapable condition: isRegBase + UB constraint ← medium
562TwoPassPerf IsCapable condition: isRegBase + rowAlign hard limit + joint threshold + UB constraint ← strictest579TwoPassPerf IsCapable condition: isRegBase + rowAlign hard limit + joint threshold + UB constraint ← strictest
563```580```
564 581 
565-The framework starts checking from Welford (most relaxed); if satisfied, it selects it directly; only if not satisfied does it try stricter branches level by level. This fully matches the design of template priorities (4000→200→150).582+The strictness order above does not represent the matching order. The framework checks in the order SplitR → TwoPassPerf → TwoPass → Welford, and the first successful template generates the tiling.
566 583 
567### 5.3 Three-Way Trade-Off Analysis: Precision, Performance, and Applicability584### 5.3 Three-Way Trade-Off Analysis: Precision, Performance, and Applicability
568 585 
@@ -574,47 +591,33 @@ No algorithm can be optimal in all three dimensions at the same time. Understand
574| **TwoPass** | Adaptability to extra-long axes | One extra global data transfer (3 times vs 2 times) | Obvious performance loss when R is small; no precision advantage |591| **TwoPass** | Adaptability to extra-long axes | One extra global data transfer (3 times vs 2 times) | Obvious performance loss when R is small; no precision advantage |
575| **TwoPassPerf** | Extreme performance + excellent precision | rowAlign and colSize/rowAlign are limited by the joint threshold | Unavailable when the shape exceeds the bounds; falls back to a slower branch |592| **TwoPassPerf** | Extreme performance + excellent precision | rowAlign and colSize/rowAlign are limited by the joint threshold | Unavailable when the shape exceeds the bounds; falls back to a slower branch |
576 593 
577-**Design philosophy:** The three algorithms do not replace each other; instead, they form a covering partition of the "**shape space**". The framework uses priorities (4000→200→150) to configure common cases optimally, and uses IsCapable conditions to guarantee a fallback in extreme cases.594+**Design philosophy:** The three original algorithms do not replace each other; instead, they form a covering partition of the "**shape space**". The framework configures common cases optimally through priorities (140→150→200→4000), and IsCapable conditions guarantee a fallback in extreme cases. SplitR increases core utilization for small A and large R, adding workspace traffic and global synchronization. Shapes outside its threshold use the original templates.
578 595 
579For readers from the software field, this "multiple algorithms covering the shape space" pattern is similar to:596For readers from the software field, this "multiple algorithms covering the shape space" pattern is similar to:
580 597 
581- **Multi-level optimization passes of compilers** — different optimization levels suit different code characteristics598- **Multi-level optimization passes of compilers** — different optimization levels suit different code characteristics
582-- **Multiple plan candidates of database query optimizers** — selecting the optimal execution plan based on a cost model599+- **Multi-plan candidates of database query optimizers** — the cost model selects the best execution plan
583-- **Kernel selection of BLAS libraries** — MKL/OpenBLAS dynamically switches between different kernels based on matrix size600+- **Kernel selection in BLAS libraries** — MKL/OpenBLAS dynamically switch kernels based on matrix size
584 601 
585----602+## 6. Source Map
586 603 
587-## 6. Quick Reference of Code Structure604+| File | Responsibility |
588- 605+| --- | --- |
589-| File | Execution Location | Content |606+| `op_host/layer_norm_v4_split_r_tiling.cpp` | SplitR admission, UB budget, workspace and scheduling |
590-|------|---------|------|607+| `op_kernel/arch35/layer_norm_v4_split_r.h` | Segmentation, publication, compact gather, merge and normalization |
591-| `op_kernel/arch35/layer_norm_v4_welford.h` | AI Core (device side) | Parallel Welford kernel implementation |608+| `op_kernel/arch35/layer_norm_v4_welford_base.h` | Shared Welford buffers and computation |
592-| `op_host/layer_norm_v4_welford_tiling.cpp` | Host CPU | Welford tiling parameter computation (including IsCapable) |609+| `op_kernel/arch35/layer_norm_v4_welford.h` | Whole-row Welford execution |
593-| `op_kernel/arch35/layer_norm_v4_two_pass.h` | AI Core (device side) | TwoPass + binary accumulation kernel implementation |610+| `op_host/layer_norm_v4_welford_tiling.cpp` | Welford tile selection |
594-| `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | Host CPU | TwoPass tiling parameter computation (including IsCapable, DoOpTiling) |611+| `op_kernel/arch35/layer_norm_v4_two_pass.h` | Two-Pass computation |
595-| `op_kernel/arch35/layer_norm_v4_two_pass_perf.h` | AI Core (device side) | TwoPass Perf optimized kernel |612+| `op_kernel/arch35/layer_norm_v4_two_pass_perf.h` | Optimized Two-Pass computation |
596-| `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Host CPU | TwoPass Perf tiling (including A/R threshold definitions in IsCapable) |613+| `op_host/layer_norm_v4_regbase_two_pass_tiling.cpp` | Two-Pass tiling |
597-| `op_kernel/layer_norm_v4_apt.h` | AI Core (device side) | 950 platform kernel entry and TILING_KEY dispatch |614+| `op_host/layer_norm_v4_regbase_two_pass_perf_tiling.cpp` | Two-Pass Perf tiling |
598-| `op_host/layer_norm_v4_tiling.h` | Host CPU | Tiling data structure definitions (shared by templates) |615+| `op_kernel/layer_norm_v4_apt.h` | Kernel dispatch |
599-| `op_host/layer_norm_v4_def.cpp:191` | Host CPU | 950 platform configuration, registered as `layer_norm_v4_apt` |616+| `op_host/layer_norm_v4_tiling.h` | Tiling data definitions |
600- 
601-Summary of code layering:
602- 
603-- The `.h` files under `op_kernel/arch35/` are **architecture-specific device-side kernel code** and run on the AI Core of Ascend 950
604-- The `.cpp`/`.h` files under `op_host/` are **Host-side control code** and run on the server CPU, responsible for parameter computation and scheduling
605-- Developing an operator requires maintaining code on both sides — the Host side does "strategy computation", and the Kernel side does "strategy execution"
606- 
607----
608 617 
609## 7. Summary618## 7. Summary
610 619 
611-The precision optimization of LayerNormV4 on the Ascend 950 platform reflects design ideas at two levels:620+Centered statistics, tree accumulation, and shape-based template selection address different numerical and performance constraints. SplitR extends this selection to small-A/large-R inputs while reusing the local Welford and normalization paths. Changes to reduction order and memory layout require precision and bounds regression on the resulting binary.
612- 
613-- **Algorithm level**: Parallel Welford integrates centering into every step through online recurrence updates, avoiding at the root the catastrophic cancellation of subtracting large numbers in the naive variance formula `E[X²] - (E[X])²`; extra-long-axis binary accumulation replaces chained addition with tree-shaped reduction, making the two operands of each addition similar in magnitude, and reduces the accumulation rounding error from O(n) to O(log n)
614- 
615-- **Engineering level**: the three algorithms are dynamically selected at compile time through the TILING_KEY mechanism, with template priorities (4000/200/150) and IsCapable conditions forming a two-level filter. The algorithms pursue precision (Welford), extreme adaptability (TwoPass), and extreme performance (TwoPassPerf) respectively, jointly covering the complete shape space from short axes to extra-long axes; fine-grained management of UB space (double buffer, tile size alignment, temporary buffer capacity validation) ensures efficient pipelining of data transfer and computation on the AI Core
616- 
617-These two algorithm ideas apply not only to LayerNormV4 but also to all Norm operators that need to reduce along extra-long axes (such as **RMSNorm**, **GroupNorm**, and **InstanceNorm**), and more broadly to high-precision floating-point reduction scenarios. They have universal reference value.
618 621 
619---622---
620 623 
@@ -0,0 +1,121 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+/*!
12+ * \file layer_norm_v4_split_r_tiling.cpp
13+ * \brief
14+ */
15+ 
16+#include <climits>
17+#include "layer_norm_v4_tiling.h"
18+ 
19+namespace optiling {
20+namespace {
21+constexpr int64_t SPLIT_R_TEMPLATE = 8;
22+// Conservative small-A/large-R dispatch policy, not hardware limits.
23+// Keep enough work per split to amortize workspace traffic and SyncAll.
24+constexpr int64_t SPLIT_R_MIN_ELEMENTS = 16384;
25+constexpr int64_t SPLIT_R_MIN_ROW = 65536;
26+constexpr int64_t SPLIT_R_MAX_CORES = 16;
27+constexpr int64_t SPLIT_R_CORES_PER_ROW = 4;
28+constexpr int64_t SPLIT_R_MIN_CORES = 2;
29+constexpr uint64_t STATISTIC_PLANES = 2;
30+} // namespace
31+ 
32+class LayerNormV4SplitRTiling : public LayerNormV4WelfordTiling {
33+public:
34+ explicit LayerNormV4SplitRTiling(gert::TilingContext* context) : LayerNormV4WelfordTiling(context) {}
35+ 
36+protected:
37+ bool IsCapable() override
38+ {
39+ // A <= floor(AIV cores / 4), 65536 <= R <= INT32_MAX.
40+ // With 64 AIV cores this admits A <= 16. Each split has >= 16384 elements.
41+ if (!commonParams.isRegBase || commonParams.isV3) {
42+ return false;
43+ }
44+ if (commonParams.colSize == 0 || commonParams.colSize > commonParams.coreNum / SPLIT_R_CORES_PER_ROW) {
45+ return false;
46+ }
47+ splitR_ = std::min<int64_t>(commonParams.coreNum / commonParams.colSize, SPLIT_R_MAX_CORES);
48+ if (commonParams.rowSize < SPLIT_R_MIN_ROW || commonParams.rowSize > INT32_MAX) {
49+ return false;
50+ }
51+ splitR_ = std::min<int64_t>(splitR_, commonParams.rowSize / SPLIT_R_MIN_ELEMENTS);
52+ return splitR_ >= SPLIT_R_MIN_CORES;
53+ }
54+ 
55+ // The UB deduction must wrap the base DoLibApiTiling call: the tileLength
56+ // selection loop (IsValidTileLength vs commonParams.ubSizePlatForm) runs
57+ // inside it, so the reserved partial-queue UB is excluded before
58+ // tileLength is chosen. The base DoOpTiling only derives the block
59+ // partitioning from M/coreNum and reads no UB budget, so it needs no
60+ // deduction and is inherited unchanged.
61+ ge::graphStatus DoLibApiTiling() override
62+ {
63+ // Separate mean/variance planes coexist with the Welford buffers.
64+ // Match the vector-aligned partial planes allocated by the kernel.
65+ uint64_t vectorBytes = Ops::Base::GetVRegSize(context_);
66+ uint64_t planeBytes = ((splitR_ * sizeof(float) + vectorBytes - 1) / vectorBytes) * vectorBytes;
67+ uint64_t reserved = STATISTIC_PLANES * planeBytes;
68+ if (commonParams.ubSizePlatForm <= reserved) {
69+ return ge::GRAPH_FAILED;
70+ }
71+ commonParams.ubSizePlatForm -= reserved;
72+ auto status = LayerNormV4WelfordTiling::DoLibApiTiling();
73+ commonParams.ubSizePlatForm += reserved;
fanzhaoyun001
fanzhaoyun001fanzhaoyun00119 days ago

[Minor] DoLibApiTiling 恢复 ubSizePlatForm 后,需确认后续阶段不会再按全量 UB 规划缓冲

这里在基类 DoLibApiTiling() 前后对 ubSizePlatForm 做「扣减 partial 队列 → 计算 tileLength → 恢复」。前提是:基类 DoOpTiling() 及后续阶段不再读取 ubSizePlatForm 规划 UB 缓冲,否则 partialIn_/partialOut_((S+1)×32B)会与基类 welfordTemp/apiTemp/输入输出队列叠加,最终超出 UB。

请确认:

  1. LayerNormV4WelfordTiling::DoOpTiling() 以及 SplitR 之后的所有 UB 分配都只依赖 tileLength(而非完整 ubSizePlatForm);
  2. reserved=(splitR_+1)*32 与 kernel 实际 partialOut_(32B) + partialIn_(S×32B) 一致(当前一致)。
likedislike
pipiman
pipiman
18 days ago Comments:
likedislike
74+ return status;
atomgit-bot
atomgit-botatomgit-bot23 days ago

🟡 Medium Priority

建议:把 UB 预留从「仅包裹基类 DoLibApiTiling」改为「覆盖 DoOpTiling 与 DoLibApiTiling」:例如重写 DoOpTiling(),在其中先扣减 (splitR_+1)*PARTIAL_RECORD_BYTES 再调用基类 DoOpTiling/DoLibApiTiling,结束后恢复,或直接返回失败,避免基类 IsValidTileLength 用未扣减的 ubSizePlatForm 选出导致 UB 超额的 tileLength。

likedislike
Incorrect?
pipiman
pipiman
22 days ago Comments:
likedislike
75+ }
76+ 
77+ uint64_t GetTilingKey() const override { return SPLIT_R_TEMPLATE * LN_TEMPLATE_KEY_WEIGHT + commonParams.dtypeKey; }
78+ 
79+ ge::graphStatus PostTiling() override
80+ {
81+ splitTd_.set_M(td_.get_M());
82+ splitTd_.set_N(td_.get_N());
83+ splitTd_.set_rAlign(td_.get_rAlign());
84+ splitTd_.set_numBlocks(td_.get_numBlocks());
85+ splitTd_.set_mainBlockCount(td_.get_mainBlockCount());
86+ splitTd_.set_mainBlockFactor(td_.get_mainBlockFactor());
87+ splitTd_.set_tailBlockFactor(td_.get_tailBlockFactor());
88+ splitTd_.set_tileLength(td_.get_tileLength());
89+ splitTd_.set_welfordTempSize(td_.get_welfordTempSize());
90+ splitTd_.set_welfordUpdateTimes(td_.get_welfordUpdateTimes());
91+ splitTd_.set_welfordUpdateTail(td_.get_welfordUpdateTail());
92+ splitTd_.set_nullptrGamma(td_.get_nullptrGamma());
93+ splitTd_.set_nullptrBeta(td_.get_nullptrBeta());
94+ splitTd_.set_apiTempBufferSize(td_.get_apiTempBufferSize());
95+ splitTd_.set_epsilon(td_.get_epsilon());
96+ splitTd_.set_splitR(splitR_);
97+ context_->SetBlockDim(commonParams.colSize * splitR_);
98+ auto* raw = context_->GetRawTilingData();
99+ if (splitTd_.GetDataSize() > raw->GetCapacity()) {
100+ return ge::GRAPH_FAILED;
101+ }
102+ splitTd_.SaveToBuffer(raw->GetData(), raw->GetCapacity());
103+ raw->SetDataSize(splitTd_.GetDataSize());
104+ auto* sizes = context_->GetWorkspaceSizes(1);
105+ auto* info = context_->GetPlatformInfo();
106+ OP_CHECK_NULL_WITH_CONTEXT(context_, sizes);
107+ OP_CHECK_NULL_WITH_CONTEXT(context_, info);
108+ platform_ascendc::PlatformAscendC platform(info);
109+ // Two statistic planes (mean and variance), independent of double buffering.
110+ sizes[0] = platform.GetLibApiWorkSpaceSize() +
111+ STATISTIC_PLANES * commonParams.colSize * splitR_ * Ops::Base::GetUbBlockSize(context_);
112+ return context_->SetScheduleMode(1U);
113+ }
114+ 
115+private:
116+ int64_t splitR_ = 1;
117+ LayerNormV4TilingDataSplitR splitTd_;
118+};
119+ 
120+REGISTER_OPS_TILING_TEMPLATE(LayerNormV4, LayerNormV4SplitRTiling, 140);
121+} // namespace optiling
@@ -131,6 +131,31 @@ REGISTER_TILING_DATA_CLASS(LayerNormV4_411, LayerNormV4TilingDataWelford)
131REGISTER_TILING_DATA_CLASS(LayerNormV4_420, LayerNormV4TilingDataWelford)131REGISTER_TILING_DATA_CLASS(LayerNormV4_420, LayerNormV4TilingDataWelford)
132REGISTER_TILING_DATA_CLASS(LayerNormV4_422, LayerNormV4TilingDataWelford)132REGISTER_TILING_DATA_CLASS(LayerNormV4_422, LayerNormV4TilingDataWelford)
133 133 
134+BEGIN_TILING_DATA_DEF(LayerNormV4TilingDataSplitR)
135+TILING_DATA_FIELD_DEF(int64_t, M); // 输入tensor的行
136+TILING_DATA_FIELD_DEF(int64_t, N); // 输入tensor的列,即reduce的轴
137+TILING_DATA_FIELD_DEF(int64_t, rAlign); // r对齐的大小
138+TILING_DATA_FIELD_DEF(int64_t, numBlocks); // 实际使用的core数量
139+TILING_DATA_FIELD_DEF(int64_t, mainBlockCount); // 整核的数量
140+TILING_DATA_FIELD_DEF(int64_t, mainBlockFactor); // 整核处理的row大小
141+TILING_DATA_FIELD_DEF(int64_t, tailBlockFactor); // 尾核处理的row大小
142+TILING_DATA_FIELD_DEF(int64_t, tileLength); // tile块的元素个数
143+TILING_DATA_FIELD_DEF(int64_t, welfordTempSize); // welford临时buffer的大小
144+TILING_DATA_FIELD_DEF(int64_t, welfordUpdateTimes); // welford update的次数
145+TILING_DATA_FIELD_DEF(int64_t, welfordUpdateTail); // welford update的尾数
146+TILING_DATA_FIELD_DEF(int64_t, nullptrGamma);
147+TILING_DATA_FIELD_DEF(int64_t, nullptrBeta);
148+TILING_DATA_FIELD_DEF(int64_t, apiTempBufferSize);
149+TILING_DATA_FIELD_DEF(float, epsilon);
150+TILING_DATA_FIELD_DEF(int64_t, splitR);
151+END_TILING_DATA_DEF;
152+ 
153+REGISTER_TILING_DATA_CLASS(LayerNormV4_800, LayerNormV4TilingDataSplitR)
154+REGISTER_TILING_DATA_CLASS(LayerNormV4_810, LayerNormV4TilingDataSplitR)
155+REGISTER_TILING_DATA_CLASS(LayerNormV4_811, LayerNormV4TilingDataSplitR)
156+REGISTER_TILING_DATA_CLASS(LayerNormV4_820, LayerNormV4TilingDataSplitR)
157+REGISTER_TILING_DATA_CLASS(LayerNormV4_822, LayerNormV4TilingDataSplitR)
158+ 
134BEGIN_TILING_DATA_DEF(LayerNormV4TilingDataRegBaseTwoPassPerf)159BEGIN_TILING_DATA_DEF(LayerNormV4TilingDataRegBaseTwoPassPerf)
135TILING_DATA_FIELD_DEF(int64_t, a);160TILING_DATA_FIELD_DEF(int64_t, a);
136TILING_DATA_FIELD_DEF(int64_t, aBlockFactor);161TILING_DATA_FIELD_DEF(int64_t, aBlockFactor);
@@ -0,0 +1,266 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+/*!
12+ * \file layer_norm_v4_split_r.h
13+ * \brief
14+ */
15+#ifndef LAYER_NORM_V4_SPLIT_R_H
16+#define LAYER_NORM_V4_SPLIT_R_H
17+ 
18+#include "layer_norm_v4_welford_base.h"
19+#include "../../norm_common/reduce_common_regbase.h"
20+ 
21+namespace LayerNormV4 {
22+// Build weighted contributions before invoking the common length-aware reduction.
23+class LayerNormV4SplitRCommon {
24+public:
25+ __aicore__ inline static void FillCountBlock(__ubuf__ float* dst, AscendC::Reg::RegTensor<float>& value,
26+ AscendC::Reg::MaskReg& full, AscendC::Reg::MaskReg& active,
27+ float count, uint32_t num, uint16_t loops, uint32_t vl)
28+ {
29+ AscendC::Reg::Duplicate(value, count, full);
30+ for (uint16_t i = 0; i < loops; ++i) {
31+ active = AscendC::Reg::UpdateMask<float>(num);
32+ AscendC::Reg::StoreAlign(dst + i * vl, value, active);
33+ }
34+ }
35+ 
36+ __aicore__ inline static void Merge(LocalTensor<float>& meanOut, LocalTensor<float>& varianceOut,
37+ LocalTensor<float>& means, LocalTensor<float>& variances,
38+ LocalTensor<float>& counts, LocalTensor<float>& contributions,
39+ LocalTensor<float>& reduceTmp, uint32_t splits, float inverseCount)
40+ {
41+ auto* meanAddr = (__ubuf__ float*)means.GetPhyAddr();
42+ auto* varAddr = (__ubuf__ float*)variances.GetPhyAddr();
43+ auto* countAddr = (__ubuf__ float*)counts.GetPhyAddr();
44+ auto* contributionAddr = (__ubuf__ float*)contributions.GetPhyAddr();
45+ auto* meanOutAddr = (__ubuf__ float*)meanOut.GetPhyAddr();
46+ constexpr uint32_t vl = NormCommon::V_LENGTH;
47+ uint32_t foldPoint = vl;
48+ while (foldPoint <= splits / 2) {
49+ foldPoint *= 2;
50+ }
51+ // RegBase vector loops require a uint16_t induction variable and a
52+ // bound divisible by the step; the mask covers the aligned tail.
53+ uint32_t splitsAligned = (splits + vl - 1) / vl * vl;
54+ __VEC_SCOPE__
55+ {
56+ using namespace AscendC::Reg;
57+ RegTensor<float> mean, weight, reference, delta;
58+ LoadAlign<float, LoadDist::DIST_BRC_B32>(reference, meanAddr);
59+ uint32_t remaining = splits;
60+ for (uint16_t offset = 0; offset < splitsAligned; offset += vl) {
61+ MaskReg active = UpdateMask<float>(remaining);
62+ LoadAlign(mean, meanAddr + offset);
63+ LoadAlign(weight, countAddr + offset);
64+ Muls(weight, weight, inverseCount, active);
65+ // Retain the centered-mean fix for large-offset inputs.
66+ Sub(delta, mean, reference, active);
67+ Mul(delta, delta, weight, active);
68+ StoreAlign(contributionAddr + offset, delta, active);
69+ }
70+ LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>();
71+ }
72+ NormCommon::NormCommonRegbase::CalculateReduceSum(contributions, meanOut, reduceTmp, splits, foldPoint);
73+ __VEC_SCOPE__
74+ {
75+ using namespace AscendC::Reg;
76+ RegTensor<float> mean, variance, weight, reference, delta, mergedMean;
77+ MaskReg first = CreateMask<float, MaskPattern::VL1>();
78+ LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>();
79+ LoadAlign<float, LoadDist::DIST_BRC_B32>(reference, meanAddr);
80+ LoadAlign<float, LoadDist::DIST_BRC_B32>(mergedMean, meanOutAddr);
81+ Add(mergedMean, mergedMean, reference, first);
82+ StoreAlign(meanOutAddr, mergedMean, first);
83+ LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>();
84+ LoadAlign<float, LoadDist::DIST_BRC_B32>(mergedMean, meanOutAddr);
85+ uint32_t remaining = splits;
86+ for (uint16_t offset = 0; offset < splitsAligned; offset += vl) {
87+ MaskReg active = UpdateMask<float>(remaining);
88+ LoadAlign(mean, meanAddr + offset);
89+ LoadAlign(variance, varAddr + offset);
90+ LoadAlign(weight, countAddr + offset);
91+ // Weight before multiplying variance to avoid count-amplified overflow.
92+ Muls(weight, weight, inverseCount, active);
93+ Sub(delta, mean, mergedMean, active);
94+ Mul(delta, delta, delta, active);
95+ Mul(delta, delta, weight, active);
96+ Mul(variance, variance, weight, active);
97+ Add(variance, variance, delta, active);
98+ StoreAlign(contributionAddr + offset, variance, active);
99+ }
100+ LocalMemBar<MemType::VEC_STORE, MemType::VEC_LOAD>();
101+ }
102+ NormCommon::NormCommonRegbase::CalculateReduceSum(contributions, varianceOut, reduceTmp, splits, foldPoint);
103+ }
104+};
105+ 
106+template <typename T, typename U, typename M>
107+class LayerNormV4RegbaseSplitR : public LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataSplitR> {
108+ using Base = LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataSplitR>;
109+ 
110+public:
111+ __aicore__ inline LayerNormV4RegbaseSplitR(const LayerNormV4TilingDataSplitR* tilingData, TPipe* pipeIn)
112+ : Base(tilingData, pipeIn)
113+ {}
114+ 
115+ __aicore__ inline void Init(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd,
116+ GM_ADDR workspace)
117+ {
118+ row_ = GetBlockIdx() / this->td_->splitR;
119+ split_ = GetBlockIdx() % this->td_->splitR;
120+ segmentStart_ = SegmentStart(split_);
121+ segmentCount_ = SegmentCount(split_);
122+ this->xGm.SetGlobalBuffer((__gm__ T*)x + row_ * this->td_->N + segmentStart_, segmentCount_);
123+ this->yGm.SetGlobalBuffer((__gm__ T*)y + row_ * this->td_->N + segmentStart_, segmentCount_);
124+ this->meanGm.SetGlobalBuffer((__gm__ M*)mean + row_, 1);
125+ this->rstdGm.SetGlobalBuffer((__gm__ M*)rstd + row_, 1);
126+ partialGm_.SetGlobalBuffer((__gm__ float*)workspace);
127+ // Each GM record occupies a UB block; compact copies pack lane 0 from
128+ // every record into vector-aligned mean/variance planes.
129+ partialHalf_ = (this->td_->splitR + VECTOR_FLOATS - 1) / VECTOR_FLOATS * VECTOR_FLOATS;
130+ this->pipe_->InitBuffer(partialIn_, 1, 2 * partialHalf_ * sizeof(float));
131+ this->InitCommon(gamma, beta);
132+ }
133+ 
134+ __aicore__ inline void Process()
135+ {
136+ this->ResetCache();
137+ this->mean_ = this->welfordTempBuffer.template Get<float>();
138+ this->variance_ = this->mean_[this->td_->tileLength];
139+ this->varianceTensor = this->variance_[this->td_->tileLength];
140+ this->shared_ = this->apiTempBuffer.template Get<uint8_t>();
141+ this->WelfordInitialize(this->mean_, this->variance_, this->td_->tileLength);
142+ int64_t loops = segmentCount_ / this->td_->tileLength;
143+ int64_t tail = segmentCount_ % this->td_->tileLength;
144+ for (int64_t i = 0; i < loops; ++i) {
145+ this->ProcessWelfordUpdate(i * this->td_->tileLength, this->td_->tileLength);
146+ }
147+ if (tail > 0) {
148+ this->ProcessWelfordUpdate(loops * this->td_->tileLength, tail);
149+ }
150+ WelfordFinalizePara para;
151+ para.rnLength = this->welfordCount;
152+ para.abLength = this->td_->tileLength;
153+ para.headCount = this->welfordCount;
154+ para.headCountLength = tail;
155+ para.tailCount = this->welfordCount - (tail > 0);
156+ para.tailCountLength = this->td_->tileLength - tail;
157+ para.abRec = 1.0f / static_cast<float>(this->td_->tileLength);
158+ para.rRec = 1.0f / static_cast<float>(segmentCount_);
159+ WelfordFinalize<true>(this->meanTensor, this->varianceTensor, this->mean_, this->variance_, this->shared_,
160+ para);
161+ 
162+ // Publish mean and population variance in separate GM planes.
163+ // SyncAll includes the preceding MTE3 writes before peer reads.
164+ auto vToMte3 = this->pipe_->FetchEventID(HardEvent::V_MTE3);
165+ SetFlag<HardEvent::V_MTE3>(vToMte3);
166+ WaitFlag<HardEvent::V_MTE3>(vToMte3);
167+ int64_t planeLength = this->td_->M * this->td_->splitR * PARTIAL_FLOATS;
168+ DataCopyExtParams partialCopyParams{1, PARTIAL_FLOATS * sizeof(float), 0, 0, 0};
169+ DataCopyPad(partialGm_[GetBlockIdx() * PARTIAL_FLOATS], this->meanTensor, partialCopyParams);
170+ DataCopyPad(partialGm_[planeLength + GetBlockIdx() * PARTIAL_FLOATS], this->varianceTensor, partialCopyParams);
171+ SyncAll();
qianzehong
qianzehongqianzehong19 days ago

[P1][ascendc-red-line][RED-7] SplitR 需要启用硬同步所需的 Kernel 类型(报告严重级别:高)

这里新增了无参数硬件 SyncAll(),但 layer_norm_v4_apt.h:110 仍只有 KERNEL_TASK_TYPE_DEFAULT(KERNEL_TYPE_AIV_ONLY),800/810/811/820/822 五个 SplitR key 均没有 MIX 类型覆盖。

DAV_3510 上 AIV_ONLY 不开启核间同步所需的调度模块;此处硬同步实际使用 ffts_cross_core_sync / wait_flag_dev,因此存在同步失效、读取其他核尚未完成的 partial,或等待同步信号挂起的风险。Host 的 SetScheduleMode(1) 只保证核调度方式,不能替代编译产物中的 Kernel 类型声明。本地 SDK 编译链也显示,显式 AIV_ONLY 会抑制硬同步的自动 MIX 推断。

建议:为这五个 SplitR key 配置 KERNEL_TASK_TYPE(key, KERNEL_TYPE_MIX_AIV_1_0),参考同仓 norm/rms_norm/op_kernel/rms_norm_apt.cpp 的 SplitAR 分支;核对实际工程编译出的 Kernel 元数据,并补充多核、多 stream 和非对齐尾段验证。

依据:官方核间同步类型约束。此结论为源码、SDK 和文档契约核实,未声称已在 NPU 实测卡死。

检视版本:5eb535c4ef9bf78b07add98e546fbb3a7a0574d1。

likedislike
pipiman
pipiman
18 days ago Comments:
likedislike
172+ 
173+ auto all = partialIn_.template AllocTensor<float>();
174+ auto allMean = all;
175+ auto allVar = all[partialHalf_];
176+ int64_t rowOffset = row_ * this->td_->splitR * PARTIAL_FLOATS;
177+ DataCopyExtParams compactCopy{static_cast<uint16_t>(this->td_->splitR), sizeof(float),
178+ static_cast<uint32_t>((PARTIAL_FLOATS - 1) * sizeof(float)), 0, 0};
179+ DataCopyPadExtParams<float> noPadding{false, 0, 0, 0};
180+ DataCopyPad<float, PaddingMode::Compact>(allMean, partialGm_[rowOffset], compactCopy, noPadding);
181+ DataCopyPad<float, PaddingMode::Compact>(allVar, partialGm_[planeLength + rowOffset], compactCopy, noPadding);
182+ partialIn_.EnQue(all);
183+ all = partialIn_.template DeQue<float>();
184+ allMean = all;
185+ allVar = all[partialHalf_];
186+ // The local Welford state is dead; reuse mean_ for compact segment counts.
187+ FillMergeCounts();
pipiman
pipimanpipiman16 days ago

这里直接紧凑型搬运过来,使用DataCopyPad的compact模式,例如S1S2,不要S1..(28b)...S2

likedislike
188+ // With splitR <= 16, the dead tile-sized M2 and 256-float rstd buffers
189+ // cover the contributions and common reduction scratch, respectively.
190+ LayerNormV4SplitRCommon::Merge(this->meanTensor, this->varianceTensor, allMean, allVar, this->mean_,
191+ this->variance_, this->rstdTensor, this->td_->splitR,
192+ 1.0f / static_cast<float>(this->td_->N));
193+ partialIn_.FreeTensor(all);
194+ for (int64_t offset = 0; offset < segmentCount_; offset += this->td_->tileLength) {
195+ int64_t length = segmentCount_ - offset;
196+ length = length < this->td_->tileLength ? length : this->td_->tileLength;
197+ this->ProcessNormalize(offset, segmentStart_ + offset, length);
198+ }
199+ if (split_ == 0) {
200+ this->cacheCount = 1;
201+ this->paramAddr = 0;
202+ this->RefreshCache();
203+ } else {
204+ this->outQueueMean.FreeTensor(this->meanTensor);
205+ this->outQueueRstd.FreeTensor(this->rstdTensor);
206+ }
207+ }
208+ 
209+private:
210+ __aicore__ inline void FillMergeCounts()
211+ {
212+ int64_t units = (this->td_->N + SPLIT_ALIGNMENT - 1) / SPLIT_ALIGNMENT;
213+ float baseCount = static_cast<float>(units / this->td_->splitR * SPLIT_ALIGNMENT);
214+ float lastCount = static_cast<float>(SegmentCount(this->td_->splitR - 1));
215+ uint32_t longCount = units % this->td_->splitR;
216+ uint32_t splitCount = this->td_->splitR;
217+ uint16_t firstLoopCount = (splitCount + VECTOR_FLOATS - 1) / VECTOR_FLOATS;
218+ uint16_t secondLoopCount = (splitCount - 1 + VECTOR_FLOATS - 1) / VECTOR_FLOATS;
219+ uint16_t thirdLoopCount = (longCount + VECTOR_FLOATS - 1) / VECTOR_FLOATS;
220+ auto* counts = (__ubuf__ float*)this->mean_.GetPhyAddr();
221+ // Overwrite order: last segment, ordinary segments, then the longer
222+ // leading segments.
223+ __VEC_SCOPE__
224+ {
225+ AscendC::Reg::RegTensor<float> value;
226+ auto mask = AscendC::Reg::CreateMask<float, AscendC::Reg::MaskPattern::ALL>();
227+ AscendC::Reg::MaskReg active;
228+ LayerNormV4SplitRCommon::FillCountBlock(counts, value, mask, active, lastCount, splitCount, firstLoopCount,
229+ VECTOR_FLOATS);
230+ LayerNormV4SplitRCommon::FillCountBlock(counts, value, mask, active, baseCount, splitCount - 1,
231+ secondLoopCount, VECTOR_FLOATS);
232+ LayerNormV4SplitRCommon::FillCountBlock(counts, value, mask, active, baseCount + SPLIT_ALIGNMENT, longCount,
233+ thirdLoopCount, VECTOR_FLOATS);
234+ }
235+ }
236+ 
237+ __aicore__ inline int64_t SegmentStart(int64_t split) const
238+ {
239+ int64_t units = (this->td_->N + SPLIT_ALIGNMENT - 1) / SPLIT_ALIGNMENT;
240+ int64_t quotient = units / this->td_->splitR;
241+ int64_t remainder = units % this->td_->splitR;
242+ return (split * quotient + (split < remainder ? split : remainder)) * SPLIT_ALIGNMENT;
243+ }
244+ 
245+ __aicore__ inline int64_t SegmentCount(int64_t split) const
246+ {
247+ int64_t end = SegmentStart(split + 1);
248+ end = end < this->td_->N ? end : this->td_->N;
249+ return end - SegmentStart(split);
250+ }
251+ 
252+ static constexpr int64_t SPLIT_ALIGNMENT = 16;
253+ static constexpr int64_t PARTIAL_FLOATS = platform::GetUbBlockSize() / sizeof(float);
254+ static constexpr int64_t VECTOR_FLOATS = LayerNormV4Regbase::GetVRegSize() / sizeof(float);
255+ int64_t partialHalf_ = 0;
256+ int64_t row_ = 0;
257+ int64_t split_ = 0;
258+ int64_t segmentStart_ = 0;
259+ int64_t segmentCount_ = 0;
260+ GlobalTensor<float> partialGm_;
261+ TQue<QuePosition::VECIN, 1> partialIn_;
262+};
263+ 
264+} // namespace LayerNormV4
265+ 
266+#endif // LAYER_NORM_V4_SPLIT_R_H
@@ -12,385 +12,108 @@
12 * \file layer_norm_v4_welford.h12 * \file layer_norm_v4_welford.h
13 * \brief13 * \brief
14 */14 */
15- 
16#ifndef LAYER_NORM_V4_WELFORD_H15#ifndef LAYER_NORM_V4_WELFORD_H
17#define LAYER_NORM_V4_WELFORD_H16#define LAYER_NORM_V4_WELFORD_H
18 17 
19-#include "layer_norm_v4_regbase_common.h"18+#include "layer_norm_v4_welford_base.h"
20-#include "kernel_tiling/kernel_tiling.h"
21-#include "kernel_operator.h"
22-#include "../../inc/platform.h"
23 19 
24namespace LayerNormV4 {20namespace LayerNormV4 {
25-using namespace AscendC;
26-using AscendC::Reg::StoreAlign;
27- 
28-namespace LayerNormV4Regbase {
29-__aicore__ inline constexpr uint32_t GetVRegSize()
30-{
31-#if __CCE_AICORE__ == 310 || __CCE_AICORE__ == 920 || __NPU_ARCH == 5102
32- return AscendC::VECTOR_REG_WIDTH;
33-#else
34- return 256U;
35-#endif
36-}
37-} // namespace LayerNormV4Regbase
38- 
39-template <typename T>
40-__aicore__ inline void CopyIn(const LocalTensor<T>& dstTensor, const GlobalTensor<T>& srcTensor, const int64_t rowSize)
41-{
42- // CopyIn
43- DataCopyExtParams params;
44- params.blockCount = 1;
45- params.blockLen = rowSize * sizeof(T);
46- DataCopyPadExtParams<T> padParams;
47- padParams.isPad = false;
48- DataCopyPad(dstTensor, srcTensor, params, padParams);
49-}
50- 
51-template <typename T>
52-__aicore__ inline void CopyOut(const GlobalTensor<T>& dstTensor, const LocalTensor<T>& srcTensor, const int64_t rowSize)
53-{
54- // CopyOut
55- DataCopyExtParams params;
56- params.blockCount = 1;
57- params.blockLen = rowSize * sizeof(T);
58- DataCopyPad(dstTensor, srcTensor, params);
59-}
60- 
61template <typename T, typename U, typename M>21template <typename T, typename U, typename M>
62-class LayerNormV4RegbaseWelford {22+class LayerNormV4RegbaseWelford : public LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataWelford> {
23+ using Base = LayerNormV4WelfordBase<T, U, M, LayerNormV4TilingDataWelford>;
24+ 
63public:25public:
64 __aicore__ inline LayerNormV4RegbaseWelford(const LayerNormV4TilingDataWelford* tilingData, TPipe* pipeIn)26 __aicore__ inline LayerNormV4RegbaseWelford(const LayerNormV4TilingDataWelford* tilingData, TPipe* pipeIn)
65- {27+ : Base(tilingData, pipeIn)
66- // Constructor28+ {}
67- pipe_ = pipeIn;
68- td_ = tilingData;
69- }
70 29 
71 __aicore__ inline void Init(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd)30 __aicore__ inline void Init(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd)
72 {31 {
73- // Init32+ if (GetBlockIdx() >= this->td_->numBlocks) {
74- if (GetBlockIdx() >= td_->numBlocks) {
75 return;33 return;
76 }34 }
77- 35+ currentBlockFactor = (GetBlockIdx() == this->td_->mainBlockCount) ? this->td_->tailBlockFactor :
78- // init global memory36+ this->td_->mainBlockFactor;
79- currentBlockFactor = (GetBlockIdx() == td_->mainBlockCount) ? td_->tailBlockFactor : td_->mainBlockFactor;37+ int64_t fmOffset = GetBlockIdx() * this->td_->mainBlockFactor * this->td_->N;
80- int64_t fmOffset = GetBlockIdx() * td_->mainBlockFactor * td_->N;38+ int64_t paramOffset = GetBlockIdx() * this->td_->mainBlockFactor;
81- int64_t paramOffset = GetBlockIdx() * td_->mainBlockFactor;39+ int64_t fmSize = currentBlockFactor * this->td_->N;
82- int64_t fmSize = currentBlockFactor * td_->N;
83 int64_t paramSize = currentBlockFactor;40 int64_t paramSize = currentBlockFactor;
84 41 
85- xGm.SetGlobalBuffer((__gm__ T*)x + fmOffset, fmSize);42+ this->xGm.SetGlobalBuffer((__gm__ T*)x + fmOffset, fmSize);
86- yGm.SetGlobalBuffer((__gm__ T*)y + fmOffset, fmSize);43+ this->yGm.SetGlobalBuffer((__gm__ T*)y + fmOffset, fmSize);
87- meanGm.SetGlobalBuffer((__gm__ M*)mean + paramOffset, paramSize);44+ this->meanGm.SetGlobalBuffer((__gm__ M*)mean + paramOffset, paramSize);
88- rstdGm.SetGlobalBuffer((__gm__ M*)rstd + paramOffset, paramSize);45+ this->rstdGm.SetGlobalBuffer((__gm__ M*)rstd + paramOffset, paramSize);
89- if (td_->nullptrGamma == 0) {46+ this->InitCommon(gamma, beta);
90- gammaGm.SetGlobalBuffer((__gm__ U*)gamma, td_->N);
91- }
92- if (td_->nullptrBeta == 0) {
93- betaGm.SetGlobalBuffer((__gm__ U*)beta, td_->N);
94- }
95- 
96- // init local memory
97- pipe_->InitBuffer(inQueueX, DOUBLE_BUFFER, td_->tileLength * sizeof(T));
98- pipe_->InitBuffer(outQueueY, DOUBLE_BUFFER, td_->tileLength * sizeof(T));
99- pipe_->InitBuffer(outQueueMean, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float));
100- pipe_->InitBuffer(outQueueRstd, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float));
101- pipe_->InitBuffer(welfordTempBuffer, td_->welfordTempSize);
102- if (td_->nullptrGamma == 0) {
103- pipe_->InitBuffer(inQueueGamma, DOUBLE_BUFFER, td_->tileLength * sizeof(U));
104- }
105- if (td_->nullptrBeta == 0) {
106- pipe_->InitBuffer(inQueueBeta, DOUBLE_BUFFER, td_->tileLength * sizeof(U));
107- }
108- pipe_->InitBuffer(apiTempBuffer, td_->apiTempBufferSize);
109 }47 }
110 48 
111 __aicore__ inline void Process()49 __aicore__ inline void Process()
112 {50 {
113 // Process51 // Process
114- if (GetBlockIdx() >= td_->numBlocks) {52+ if (GetBlockIdx() >= this->td_->numBlocks) {
115 return;53 return;
116 }54 }
117 55 
118- ResetCache();56+ this->ResetCache();
119- paramAddr = 0;57+ this->paramAddr = 0;
120- mean_ = welfordTempBuffer.Get<float>();58+ this->mean_ = this->welfordTempBuffer.template Get<float>();
121- variance_ = mean_[td_->tileLength];59+ this->variance_ = this->mean_[this->td_->tileLength];
122- varianceTensor = variance_[td_->tileLength];60+ this->varianceTensor = this->variance_[this->td_->tileLength];
123- shared_ = apiTempBuffer.Get<uint8_t>();61+ this->shared_ = this->apiTempBuffer.template Get<uint8_t>();
124 62 
125 for (int64_t i = 0; i < currentBlockFactor; ++i) {63 for (int64_t i = 0; i < currentBlockFactor; ++i) {
126 // Welford Algorithm64 // Welford Algorithm
127- WelfordInitialize(mean_, variance_, td_->tileLength);65+ this->WelfordInitialize(this->mean_, this->variance_, this->td_->tileLength);
128- for (int64_t welfordUpdateCount = 0; welfordUpdateCount < td_->welfordUpdateTimes; welfordUpdateCount++) {66+ for (int64_t welfordUpdateCount = 0; welfordUpdateCount < this->td_->welfordUpdateTimes;
129- int64_t offset = i * td_->N + welfordUpdateCount * td_->tileLength;67+ welfordUpdateCount++) {
130- ProcessWelfordUpdate(offset, td_->tileLength);68+ int64_t offset = i * this->td_->N + welfordUpdateCount * this->td_->tileLength;
69+ this->ProcessWelfordUpdate(offset, this->td_->tileLength);
131 }70 }
132- if (td_->welfordUpdateTail > 0) {71+ if (this->td_->welfordUpdateTail > 0) {
133- int64_t offset = i * td_->N + td_->welfordUpdateTimes * td_->tileLength;72+ int64_t offset = i * this->td_->N + this->td_->welfordUpdateTimes * this->td_->tileLength;
134- ProcessWelfordUpdate(offset, td_->welfordUpdateTail);73+ this->ProcessWelfordUpdate(offset, this->td_->welfordUpdateTail);
135 }74 }
136 75 
137 WelfordFinalizePara para;76 WelfordFinalizePara para;
138- para.rnLength = welfordCount;77+ para.rnLength = this->welfordCount;
139- para.abLength = td_->tileLength;78+ para.abLength = this->td_->tileLength;
140- para.headCount = welfordCount;79+ para.headCount = this->welfordCount;
141- para.headCountLength = td_->welfordUpdateTail;80+ para.headCountLength = this->td_->welfordUpdateTail;
142- para.tailCount = welfordCount - (td_->welfordUpdateTail > 0);81+ para.tailCount = this->welfordCount - (this->td_->welfordUpdateTail > 0);
143- para.tailCountLength = td_->tileLength - td_->welfordUpdateTail;82+ para.tailCountLength = this->td_->tileLength - this->td_->welfordUpdateTail;
144- para.abRec = 1.0f / static_cast<float>(td_->tileLength);83+ para.abRec = 1.0f / static_cast<float>(this->td_->tileLength);
145- para.rRec = 1.0f / static_cast<float>(td_->N);84+ para.rRec = 1.0f / static_cast<float>(this->td_->N);
146- WelfordFinalize<true>(meanTensor[cacheCount], varianceTensor[cacheCount], mean_, variance_, shared_, para);85+ WelfordFinalize<true>(this->meanTensor[this->cacheCount], this->varianceTensor[this->cacheCount],
86+ this->mean_, this->variance_, this->shared_, para);
147 87 
148 // Normalize88 // Normalize
149- for (int64_t welfordUpdateCount = 0; welfordUpdateCount < td_->welfordUpdateTimes; welfordUpdateCount++) {89+ for (int64_t welfordUpdateCount = 0; welfordUpdateCount < this->td_->welfordUpdateTimes;
150- int64_t fmOffset = i * td_->N + welfordUpdateCount * td_->tileLength;90+ welfordUpdateCount++) {
151- int64_t paramOffset = welfordUpdateCount * td_->tileLength;91+ int64_t fmOffset = i * this->td_->N + welfordUpdateCount * this->td_->tileLength;
152- ProcessNormalize(fmOffset, paramOffset, td_->tileLength);92+ int64_t paramOffset = welfordUpdateCount * this->td_->tileLength;
93+ this->ProcessNormalize(fmOffset, paramOffset, this->td_->tileLength);
153 }94 }
154- if (td_->welfordUpdateTail > 0) {95+ if (this->td_->welfordUpdateTail > 0) {
155- int64_t fmOffset = i * td_->N + td_->welfordUpdateTimes * td_->tileLength;96+ int64_t fmOffset = i * this->td_->N + this->td_->welfordUpdateTimes * this->td_->tileLength;
156- int64_t paramOffset = td_->welfordUpdateTimes * td_->tileLength;97+ int64_t paramOffset = this->td_->welfordUpdateTimes * this->td_->tileLength;
157- ProcessNormalize(fmOffset, paramOffset, td_->welfordUpdateTail);98+ this->ProcessNormalize(fmOffset, paramOffset, this->td_->welfordUpdateTail);
158 }99 }
159 100 
160- cacheCount++;101+ this->cacheCount++;
161 // check cache buffer102 // check cache buffer
162- if (cacheCount >= AGGREGATION_COUNT) {103+ if (this->cacheCount >= this->AGGREGATION_COUNT) {
163- RefreshCache();104+ this->RefreshCache();
164- ResetCache();105+ this->ResetCache();
165 }106 }
166 }107 }
167 108 
168 // refresh cache109 // refresh cache
169- if (cacheCount > 0) {110+ if (this->cacheCount > 0) {
170- RefreshCache();111+ this->RefreshCache();
171 }112 }
172 }113 }
173 114 
174private:115private:
175- __aicore__ inline void ResetCache()
176- {
177- // ResetCache
178- cacheCount = 0;
179- meanTensor = outQueueMean.template AllocTensor<float>();
180- rstdTensor = outQueueRstd.template AllocTensor<float>();
181- }
182- 
183- __aicore__ inline void RefreshCache()
184- {
185- if constexpr (!IsSameType<M, float>::value) {
186- // float to bfloat16 or float16, input continue and output each repeat have only half value
187- CastBatchMeanRstdToDtype<M>((__ubuf__ float*)meanTensor.GetPhyAddr(),
188- (__ubuf__ float*)rstdTensor.GetPhyAddr(), (__ubuf__ M*)meanTensor.GetPhyAddr(),
189- (__ubuf__ M*)rstdTensor.GetPhyAddr(), cacheCount);
190- outQueueMean.EnQue(meanTensor);
191- outQueueRstd.EnQue(rstdTensor);
192- meanTensor = outQueueMean.template DeQue<float>();
193- rstdTensor = outQueueRstd.template DeQue<float>();
194- 
195- // VL_F32
196- uint32_t castDmaCount = static_cast<uint32_t>(cacheCount);
197- uint32_t castDmaLoops = static_cast<uint32_t>(castDmaCount / VL_F32);
198- if (castDmaLoops > 0) {
199- DataCopyExtParams copyInParams;
200- copyInParams.blockCount = castDmaLoops;
201- copyInParams.blockLen = VL_F32 * sizeof(M);
202- copyInParams.srcStride = (VECTOR_REG_WIDTH - VL_F32 * sizeof(M)) / BLOCK_SIZE;
203- copyInParams.dstStride = 0;
204- DataCopyPad(meanGm[paramAddr], meanTensor.ReinterpretCast<M>(), copyInParams);
205- DataCopyPad(rstdGm[paramAddr], rstdTensor.ReinterpretCast<M>(), copyInParams);
206- }
207- 
208- // tail
209- uint32_t tailSize = static_cast<uint32_t>(castDmaCount % VL_F32);
210- if (tailSize > 0) {
211- DataCopyExtParams copyInParamsTail;
212- copyInParamsTail.blockCount = 1;
213- copyInParamsTail.blockLen = tailSize * sizeof(M);
214- copyInParamsTail.srcStride = 0;
215- copyInParamsTail.dstStride = 0;
216- DataCopyPad(meanGm[paramAddr + castDmaLoops * VL_F32],
217- meanTensor[castDmaLoops * VL_MEAN].ReinterpretCast<M>(), copyInParamsTail);
218- DataCopyPad(rstdGm[paramAddr + castDmaLoops * VL_F32],
219- rstdTensor[castDmaLoops * VL_MEAN].ReinterpretCast<M>(), copyInParamsTail);
220- }
221- outQueueMean.FreeTensor(meanTensor);
222- outQueueRstd.FreeTensor(rstdTensor);
223- } else {
224- // Refresh Cache
225- outQueueMean.EnQue(meanTensor);
226- meanTensor = outQueueMean.template DeQue<float>();
227- CopyOut(meanGm[paramAddr], meanTensor, cacheCount);
228- outQueueMean.FreeTensor(meanTensor);
229- 
230- outQueueRstd.EnQue(rstdTensor);
231- rstdTensor = outQueueRstd.template DeQue<float>();
232- CopyOut(rstdGm[paramAddr], rstdTensor, cacheCount);
233- outQueueRstd.FreeTensor(rstdTensor);
234- }
235- paramAddr += cacheCount;
236- }
237- 
238- __aicore__ inline void WelfordInitialize(const LocalTensor<float>& mean, const LocalTensor<float>& variance,
239- const int64_t elemCnt)
240- {
241- // WelfordInitialize
242- welfordCount = 0;
243- constexpr static uint32_t VL_B32 = LayerNormV4Regbase::GetVRegSize() / sizeof(float);
244- uint16_t loopTimes = (elemCnt + VL_B32 - 1) / VL_B32;
245- __VEC_SCOPE__
246- {
247- __ubuf__ float* meamPtr = (__ubuf__ float*)mean.GetPhyAddr();
248- __ubuf__ float* variancePtr = (__ubuf__ float*)variance.GetPhyAddr();
249- uint32_t count = static_cast<uint32_t>(elemCnt);
250- AscendC::Reg::RegTensor<float> xReg;
251- AscendC::Reg::MaskReg pMask;
252- Duplicate(xReg, 0.0f);
253- for (uint16_t i = 0; i < loopTimes; ++i) {
254- pMask = AscendC::Reg::UpdateMask<float>(count);
255- StoreAlign((__ubuf__ float*)meamPtr + i * VL_B32, xReg, pMask);
256- StoreAlign((__ubuf__ float*)variancePtr + i * VL_B32, xReg, pMask);
257- }
258- }
259- }
260- 
261- __aicore__ inline void ProcessWelfordUpdate(const int64_t offset, const int64_t elemCnt)
262- {
263- // ProcessWelfordUpdate
264- welfordCount++;
265- xTensor = inQueueX.template AllocTensor<T>();
266- CopyIn(xTensor, xGm[offset], elemCnt);
267- inQueueX.EnQue(xTensor);
268- xTensor = inQueueX.template DeQue<T>();
269- 
270- WelfordUpdateParam para;
271- para.rnLength = 1;
272- para.abLength = td_->tileLength;
273- para.abComputeLength = elemCnt;
274- para.nRec = 1.0f / static_cast<float>(welfordCount);
275- WelfordUpdate<T, float, false>(mean_, variance_, mean_, variance_, xTensor, shared_, para);
276- 
277- inQueueX.FreeTensor(xTensor);
278- }
279- 
280- __aicore__ inline void ProcessNormalize(const int64_t fmOffset, const int64_t paramOffset, const int64_t elemCnt)
281- {
282- // ProcessNormalize
283- xTensor = inQueueX.template AllocTensor<T>();
284- CopyIn(xTensor, xGm[fmOffset], elemCnt);
285- inQueueX.EnQue(xTensor);
286- xTensor = inQueueX.template DeQue<T>();
287- 
288- if (td_->nullptrGamma == 0) {
289- gammaTensor = inQueueGamma.template AllocTensor<U>();
290- CopyIn(gammaTensor, gammaGm[paramOffset], elemCnt);
291- inQueueGamma.EnQue(gammaTensor);
292- gammaTensor = inQueueGamma.template DeQue<U>();
293- }
294- if (td_->nullptrBeta == 0) {
295- betaTensor = inQueueBeta.template AllocTensor<U>();
296- CopyIn(betaTensor, betaGm[paramOffset], elemCnt);
297- inQueueBeta.EnQue(betaTensor);
298- betaTensor = inQueueBeta.template DeQue<U>();
299- }
300- 
301- yTensor = outQueueY.template AllocTensor<T>();
302- 
303- NormalizePara para;
304- para.aLength = 1;
305- para.rLength = elemCnt;
306- para.rLengthWithPadding = elemCnt;
307- 
308- if (td_->nullptrGamma == 0 && td_->nullptrBeta == 0) {
309- constexpr static NormalizeConfig hasGammaBetaNormalizeConfig = {ReducePattern::AR, -1, false, false, false};
310- Normalize<U, T, false, hasGammaBetaNormalizeConfig>(yTensor, rstdTensor[cacheCount], meanTensor[cacheCount],
311- varianceTensor[cacheCount], xTensor, gammaTensor,
312- betaTensor, shared_, td_->epsilon, para);
313- } else if (td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) {
314- constexpr static NormalizeConfig hasGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, false,
315- false};
316- Normalize<U, T, false, hasGammaNoBetaNormalizeConfig>(
317- yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor,
318- gammaTensor, betaTensor, shared_, td_->epsilon, para);
319- } else if (!td_->nullptrGamma == 0 && td_->nullptrBeta == 0) {
320- constexpr static NormalizeConfig noGammaHasBetaNormalizeConfig = {ReducePattern::AR, -1, false, true,
321- false};
322- Normalize<U, T, false, noGammaHasBetaNormalizeConfig>(
323- yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor,
324- gammaTensor, betaTensor, shared_, td_->epsilon, para);
325- } else if (!td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) {
326- constexpr static NormalizeConfig noGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, true, false};
327- Normalize<U, T, false, noGammaNoBetaNormalizeConfig>(
328- yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor,
329- gammaTensor, betaTensor, shared_, td_->epsilon, para);
330- }
331- 
332- inQueueX.FreeTensor(xTensor);
333- if (td_->nullptrGamma == 0) {
334- inQueueGamma.FreeTensor(gammaTensor);
335- }
336- if (td_->nullptrBeta == 0) {
337- inQueueBeta.FreeTensor(betaTensor);
338- }
339- 
340- outQueueY.EnQue(yTensor);
341- yTensor = outQueueY.template DeQue<T>();
342- CopyOut(yGm[fmOffset], yTensor, elemCnt);
343- outQueueY.FreeTensor(yTensor);
344- }
345- 
346-private:
347- TPipe* pipe_;
348- const LayerNormV4TilingDataWelford* __restrict td_;
349- 
350- // Constants
351- constexpr static int64_t DOUBLE_BUFFER = 2;
352- constexpr static int64_t AGGREGATION_COUNT = 256;
353- constexpr static uint32_t VL_F32 = VECTOR_REG_WIDTH / sizeof(float);
354- constexpr static uint32_t VL_MEAN = VECTOR_REG_WIDTH / sizeof(M);
355- constexpr static int64_t BLOCK_SIZE = platform::GetUbBlockSize();
356- 
357- // TQue
358- TQue<QuePosition::VECIN, 1> inQueueX;
359- TQue<QuePosition::VECIN, 1> inQueueGamma;
360- TQue<QuePosition::VECIN, 1> inQueueBeta;
361- 
362- TQue<QuePosition::VECOUT, 1> outQueueY;
363- TQue<QuePosition::VECOUT, 1> outQueueMean;
364- TQue<QuePosition::VECOUT, 1> outQueueRstd;
365- 
366- TBuf<TPosition::VECCALC> welfordTempBuffer;
367- TBuf<TPosition::VECCALC> apiTempBuffer;
368- 
369- // Global Tensor
370- GlobalTensor<T> xGm;
371- GlobalTensor<T> yGm;
372- GlobalTensor<M> meanGm;
373- GlobalTensor<M> rstdGm;
374- GlobalTensor<U> gammaGm;
375- GlobalTensor<U> betaGm;
376- 
377- // Local Tensor
378- LocalTensor<T> xTensor;
379- LocalTensor<T> yTensor;
380- LocalTensor<U> gammaTensor;
381- LocalTensor<U> betaTensor;
382- LocalTensor<float> meanTensor;
383- LocalTensor<float> rstdTensor;
384- LocalTensor<float> varianceTensor;
385- LocalTensor<float> mean_;
386- LocalTensor<float> variance_;
387- LocalTensor<uint8_t> shared_;
388- 
389- // params
390 int64_t currentBlockFactor = 0;116 int64_t currentBlockFactor = 0;
391- int64_t cacheCount = 0;
392- int64_t paramAddr = 0;
393- int64_t welfordCount = 0;
394};117};
395 118 
396} // namespace LayerNormV4119} // namespace LayerNormV4
@@ -0,0 +1,317 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+/*!
12+ * \file layer_norm_v4_welford_base.h
13+ * \brief
14+ */
15+ 
16+#ifndef LAYER_NORM_V4_WELFORD_BASE_H
17+#define LAYER_NORM_V4_WELFORD_BASE_H
18+ 
19+#include "layer_norm_v4_regbase_common.h"
20+#include "kernel_tiling/kernel_tiling.h"
21+#include "kernel_operator.h"
22+#include "../../inc/platform.h"
23+ 
24+namespace LayerNormV4 {
25+using namespace AscendC;
26+using AscendC::Reg::StoreAlign;
27+ 
28+namespace LayerNormV4Regbase {
29+__aicore__ inline constexpr uint32_t GetVRegSize()
30+{
31+#if __CCE_AICORE__ == 310 || __CCE_AICORE__ == 920 || __NPU_ARCH == 5102
32+ return AscendC::VECTOR_REG_WIDTH;
33+#else
34+ return 256U;
35+#endif
36+}
37+} // namespace LayerNormV4Regbase
38+ 
39+template <typename T>
40+__aicore__ inline void CopyIn(const LocalTensor<T>& dstTensor, const GlobalTensor<T>& srcTensor, const int64_t rowSize)
41+{
42+ // CopyIn
43+ DataCopyExtParams params;
44+ params.blockCount = 1;
45+ params.blockLen = rowSize * sizeof(T);
46+ DataCopyPadExtParams<T> padParams;
47+ padParams.isPad = false;
48+ DataCopyPad(dstTensor, srcTensor, params, padParams);
49+}
50+ 
51+template <typename T>
52+__aicore__ inline void CopyOut(const GlobalTensor<T>& dstTensor, const LocalTensor<T>& srcTensor, const int64_t rowSize)
53+{
54+ // CopyOut
55+ DataCopyExtParams params;
56+ params.blockCount = 1;
57+ params.blockLen = rowSize * sizeof(T);
58+ DataCopyPad(dstTensor, srcTensor, params);
59+}
60+ 
61+// Shared buffers and tile computation; execution scheduling belongs to derived kernels.
62+template <typename T, typename U, typename M, typename TilingData>
63+class LayerNormV4WelfordBase {
64+protected:
65+ __aicore__ inline LayerNormV4WelfordBase(const TilingData* tilingData, TPipe* pipeIn)
66+ : pipe_(pipeIn), td_(tilingData)
67+ {}
68+ 
69+ __aicore__ inline void InitCommon(GM_ADDR gamma, GM_ADDR beta)
70+ {
71+ if (td_->nullptrGamma == 0) {
72+ gammaGm.SetGlobalBuffer((__gm__ U*)gamma, td_->N);
73+ }
74+ if (td_->nullptrBeta == 0) {
75+ betaGm.SetGlobalBuffer((__gm__ U*)beta, td_->N);
76+ }
77+ 
78+ // init local memory
79+ pipe_->InitBuffer(inQueueX, DOUBLE_BUFFER, td_->tileLength * sizeof(T));
80+ pipe_->InitBuffer(outQueueY, DOUBLE_BUFFER, td_->tileLength * sizeof(T));
81+ pipe_->InitBuffer(outQueueMean, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float));
82+ pipe_->InitBuffer(outQueueRstd, DOUBLE_BUFFER, AGGREGATION_COUNT * sizeof(float));
83+ pipe_->InitBuffer(welfordTempBuffer, td_->welfordTempSize);
84+ if (td_->nullptrGamma == 0) {
85+ pipe_->InitBuffer(inQueueGamma, DOUBLE_BUFFER, td_->tileLength * sizeof(U));
86+ }
87+ if (td_->nullptrBeta == 0) {
88+ pipe_->InitBuffer(inQueueBeta, DOUBLE_BUFFER, td_->tileLength * sizeof(U));
89+ }
90+ pipe_->InitBuffer(apiTempBuffer, td_->apiTempBufferSize);
91+ }
92+ 
93+ __aicore__ inline void ResetCache()
94+ {
95+ // ResetCache
96+ cacheCount = 0;
97+ meanTensor = outQueueMean.template AllocTensor<float>();
98+ rstdTensor = outQueueRstd.template AllocTensor<float>();
99+ }
100+ 
101+ __aicore__ inline void RefreshCache()
102+ {
103+ if constexpr (!IsSameType<M, float>::value) {
104+ // float to bfloat16 or float16, input continue and output each repeat have only half value
105+ CastBatchMeanRstdToDtype<M>((__ubuf__ float*)meanTensor.GetPhyAddr(),
106+ (__ubuf__ float*)rstdTensor.GetPhyAddr(), (__ubuf__ M*)meanTensor.GetPhyAddr(),
107+ (__ubuf__ M*)rstdTensor.GetPhyAddr(), cacheCount);
108+ outQueueMean.EnQue(meanTensor);
109+ outQueueRstd.EnQue(rstdTensor);
110+ meanTensor = outQueueMean.template DeQue<float>();
111+ rstdTensor = outQueueRstd.template DeQue<float>();
112+ 
113+ // VL_F32
114+ uint32_t castDmaCount = static_cast<uint32_t>(cacheCount);
115+ uint32_t castDmaLoops = static_cast<uint32_t>(castDmaCount / VL_F32);
116+ if (castDmaLoops > 0) {
117+ DataCopyExtParams copyInParams;
118+ copyInParams.blockCount = castDmaLoops;
119+ copyInParams.blockLen = VL_F32 * sizeof(M);
120+ copyInParams.srcStride = (VECTOR_REG_WIDTH - VL_F32 * sizeof(M)) / BLOCK_SIZE;
121+ copyInParams.dstStride = 0;
122+ DataCopyPad(meanGm[paramAddr], meanTensor.ReinterpretCast<M>(), copyInParams);
123+ DataCopyPad(rstdGm[paramAddr], rstdTensor.ReinterpretCast<M>(), copyInParams);
124+ }
125+ 
126+ // tail
127+ uint32_t tailSize = static_cast<uint32_t>(castDmaCount % VL_F32);
128+ if (tailSize > 0) {
129+ DataCopyExtParams copyInParamsTail;
130+ copyInParamsTail.blockCount = 1;
131+ copyInParamsTail.blockLen = tailSize * sizeof(M);
132+ copyInParamsTail.srcStride = 0;
133+ copyInParamsTail.dstStride = 0;
134+ // meanTensor is a LocalTensor<float> and the cast output packs each
135+ // repeat slot at VECTOR_REG_WIDTH bytes, so the tail must be indexed in
136+ // float units (castDmaLoops * VL_F32), matching the GM offset above.
137+ DataCopyPad(meanGm[paramAddr + castDmaLoops * VL_F32],
138+ meanTensor[castDmaLoops * VL_F32].ReinterpretCast<M>(), copyInParamsTail);
139+ DataCopyPad(rstdGm[paramAddr + castDmaLoops * VL_F32],
140+ rstdTensor[castDmaLoops * VL_F32].ReinterpretCast<M>(), copyInParamsTail);
141+ }
142+ outQueueMean.FreeTensor(meanTensor);
143+ outQueueRstd.FreeTensor(rstdTensor);
144+ } else {
145+ // Refresh Cache
146+ outQueueMean.EnQue(meanTensor);
147+ meanTensor = outQueueMean.template DeQue<float>();
148+ CopyOut(meanGm[paramAddr], meanTensor, cacheCount);
149+ outQueueMean.FreeTensor(meanTensor);
150+ 
151+ outQueueRstd.EnQue(rstdTensor);
152+ rstdTensor = outQueueRstd.template DeQue<float>();
153+ CopyOut(rstdGm[paramAddr], rstdTensor, cacheCount);
154+ outQueueRstd.FreeTensor(rstdTensor);
155+ }
156+ paramAddr += cacheCount;
157+ }
158+ 
159+ __aicore__ inline void WelfordInitialize(const LocalTensor<float>& mean, const LocalTensor<float>& variance,
160+ const int64_t elemCnt)
161+ {
162+ // WelfordInitialize
163+ welfordCount = 0;
164+ constexpr static uint32_t VL_B32 = LayerNormV4Regbase::GetVRegSize() / sizeof(float);
165+ uint16_t loopTimes = (elemCnt + VL_B32 - 1) / VL_B32;
166+ __VEC_SCOPE__
167+ {
168+ __ubuf__ float* meamPtr = (__ubuf__ float*)mean.GetPhyAddr();
169+ __ubuf__ float* variancePtr = (__ubuf__ float*)variance.GetPhyAddr();
170+ uint32_t count = static_cast<uint32_t>(elemCnt);
171+ AscendC::Reg::RegTensor<float> xReg;
172+ AscendC::Reg::MaskReg pMask;
173+ Duplicate(xReg, 0.0f);
174+ for (uint16_t i = 0; i < loopTimes; ++i) {
175+ pMask = AscendC::Reg::UpdateMask<float>(count);
176+ StoreAlign((__ubuf__ float*)meamPtr + i * VL_B32, xReg, pMask);
177+ StoreAlign((__ubuf__ float*)variancePtr + i * VL_B32, xReg, pMask);
178+ }
179+ }
180+ }
181+ 
182+ __aicore__ inline void ProcessWelfordUpdate(const int64_t offset, const int64_t elemCnt)
183+ {
184+ // ProcessWelfordUpdate
185+ welfordCount++;
186+ xTensor = inQueueX.template AllocTensor<T>();
187+ CopyIn(xTensor, xGm[offset], elemCnt);
188+ inQueueX.EnQue(xTensor);
189+ xTensor = inQueueX.template DeQue<T>();
190+ 
191+ WelfordUpdateParam para;
192+ para.rnLength = 1;
193+ para.abLength = td_->tileLength;
194+ para.abComputeLength = elemCnt;
195+ para.nRec = 1.0f / static_cast<float>(welfordCount);
196+ WelfordUpdate<T, float, false>(mean_, variance_, mean_, variance_, xTensor, shared_, para);
197+ 
198+ inQueueX.FreeTensor(xTensor);
199+ }
200+ 
201+ __aicore__ inline void ProcessNormalize(const int64_t fmOffset, const int64_t paramOffset, const int64_t elemCnt)
202+ {
203+ // ProcessNormalize
204+ xTensor = inQueueX.template AllocTensor<T>();
205+ CopyIn(xTensor, xGm[fmOffset], elemCnt);
206+ inQueueX.EnQue(xTensor);
207+ xTensor = inQueueX.template DeQue<T>();
208+ 
209+ if (td_->nullptrGamma == 0) {
210+ gammaTensor = inQueueGamma.template AllocTensor<U>();
211+ CopyIn(gammaTensor, gammaGm[paramOffset], elemCnt);
212+ inQueueGamma.EnQue(gammaTensor);
213+ gammaTensor = inQueueGamma.template DeQue<U>();
214+ }
215+ if (td_->nullptrBeta == 0) {
216+ betaTensor = inQueueBeta.template AllocTensor<U>();
217+ CopyIn(betaTensor, betaGm[paramOffset], elemCnt);
218+ inQueueBeta.EnQue(betaTensor);
219+ betaTensor = inQueueBeta.template DeQue<U>();
220+ }
221+ 
222+ yTensor = outQueueY.template AllocTensor<T>();
223+ 
224+ NormalizePara para;
225+ para.aLength = 1;
226+ para.rLength = elemCnt;
227+ para.rLengthWithPadding = elemCnt;
228+ 
229+ if (td_->nullptrGamma == 0 && td_->nullptrBeta == 0) {
230+ constexpr static NormalizeConfig hasGammaBetaNormalizeConfig = {ReducePattern::AR, -1, false, false, false};
231+ Normalize<U, T, false, hasGammaBetaNormalizeConfig>(yTensor, rstdTensor[cacheCount], meanTensor[cacheCount],
232+ varianceTensor[cacheCount], xTensor, gammaTensor,
233+ betaTensor, shared_, td_->epsilon, para);
234+ } else if (td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) {
235+ constexpr static NormalizeConfig hasGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, false,
236+ false};
237+ Normalize<U, T, false, hasGammaNoBetaNormalizeConfig>(
238+ yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor,
239+ gammaTensor, betaTensor, shared_, td_->epsilon, para);
240+ } else if (!td_->nullptrGamma == 0 && td_->nullptrBeta == 0) {
241+ constexpr static NormalizeConfig noGammaHasBetaNormalizeConfig = {ReducePattern::AR, -1, false, true,
242+ false};
243+ Normalize<U, T, false, noGammaHasBetaNormalizeConfig>(
244+ yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor,
245+ gammaTensor, betaTensor, shared_, td_->epsilon, para);
246+ } else if (!td_->nullptrGamma == 0 && !td_->nullptrBeta == 0) {
247+ constexpr static NormalizeConfig noGammaNoBetaNormalizeConfig = {ReducePattern::AR, -1, true, true, false};
248+ Normalize<U, T, false, noGammaNoBetaNormalizeConfig>(
249+ yTensor, rstdTensor[cacheCount], meanTensor[cacheCount], varianceTensor[cacheCount], xTensor,
250+ gammaTensor, betaTensor, shared_, td_->epsilon, para);
251+ }
252+ 
253+ inQueueX.FreeTensor(xTensor);
254+ if (td_->nullptrGamma == 0) {
255+ inQueueGamma.FreeTensor(gammaTensor);
256+ }
257+ if (td_->nullptrBeta == 0) {
258+ inQueueBeta.FreeTensor(betaTensor);
259+ }
260+ 
261+ outQueueY.EnQue(yTensor);
262+ yTensor = outQueueY.template DeQue<T>();
263+ CopyOut(yGm[fmOffset], yTensor, elemCnt);
264+ outQueueY.FreeTensor(yTensor);
265+ }
266+ 
267+ TPipe* pipe_;
268+ const TilingData* __restrict td_;
269+ 
270+ // Constants
271+ constexpr static int64_t DOUBLE_BUFFER = 2;
272+ constexpr static int64_t AGGREGATION_COUNT = 256;
273+ constexpr static uint32_t VL_F32 = VECTOR_REG_WIDTH / sizeof(float);
274+ constexpr static uint32_t VL_MEAN = VECTOR_REG_WIDTH / sizeof(M);
275+ constexpr static int64_t BLOCK_SIZE = platform::GetUbBlockSize();
276+ 
277+ // TQue
278+ TQue<QuePosition::VECIN, 1> inQueueX;
279+ TQue<QuePosition::VECIN, 1> inQueueGamma;
280+ TQue<QuePosition::VECIN, 1> inQueueBeta;
281+ 
282+ TQue<QuePosition::VECOUT, 1> outQueueY;
283+ TQue<QuePosition::VECOUT, 1> outQueueMean;
284+ TQue<QuePosition::VECOUT, 1> outQueueRstd;
285+ 
286+ TBuf<TPosition::VECCALC> welfordTempBuffer;
287+ TBuf<TPosition::VECCALC> apiTempBuffer;
288+ 
289+ // Global Tensor
290+ GlobalTensor<T> xGm;
291+ GlobalTensor<T> yGm;
292+ GlobalTensor<M> meanGm;
293+ GlobalTensor<M> rstdGm;
294+ GlobalTensor<U> gammaGm;
295+ GlobalTensor<U> betaGm;
296+ 
297+ // Local Tensor
298+ LocalTensor<T> xTensor;
299+ LocalTensor<T> yTensor;
300+ LocalTensor<U> gammaTensor;
301+ LocalTensor<U> betaTensor;
302+ LocalTensor<float> meanTensor;
303+ LocalTensor<float> rstdTensor;
304+ LocalTensor<float> varianceTensor;
305+ LocalTensor<float> mean_;
306+ LocalTensor<float> variance_;
307+ LocalTensor<uint8_t> shared_;
308+ 
309+ // params
310+ int64_t cacheCount = 0;
311+ int64_t paramAddr = 0;
312+ int64_t welfordCount = 0;
313+};
314+ 
315+} // namespace LayerNormV4
316+ 
317+#endif // LAYER_NORM_V4_WELFORD_BASE_H
@@ -19,6 +19,7 @@
19#include "kernel_operator.h"19#include "kernel_operator.h"
20#include "arch35/layer_norm_v4_two_pass.h"20#include "arch35/layer_norm_v4_two_pass.h"
21#include "arch35/layer_norm_v4_welford.h"21#include "arch35/layer_norm_v4_welford.h"
22+#include "arch35/layer_norm_v4_split_r.h"
22#include "arch35/layer_norm_v4_two_pass_perf.h"23#include "arch35/layer_norm_v4_two_pass_perf.h"
23 24 
24using namespace LayerNormV4;25using namespace LayerNormV4;
@@ -41,6 +42,12 @@ using AscendC::AIC;
41#define LNV4_REGBASE_TWO_PASS_PERF_BF16_FLOAT 52042#define LNV4_REGBASE_TWO_PASS_PERF_BF16_FLOAT 520
42#define LNV4_REGBASE_TWO_PASS_PERF_BF16_BF16 52243#define LNV4_REGBASE_TWO_PASS_PERF_BF16_BF16 522
43 44 
45+#define LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT 800
46+#define LNV4_REGBASE_SPLIT_R_HALF_FLOAT 810
47+#define LNV4_REGBASE_SPLIT_R_HALF_HALF 811
48+#define LNV4_REGBASE_SPLIT_R_BF16_FLOAT 820
49+#define LNV4_REGBASE_SPLIT_R_BF16_BF16 822
50+ 
44template <typename Tfm, typename Tweight, typename Tmean>51template <typename Tfm, typename Tweight, typename Tmean>
45__aicore__ inline void RegbaseTwoPassImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd,52__aicore__ inline void RegbaseTwoPassImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd,
46 GM_ADDR tiling)53 GM_ADDR tiling)
@@ -76,6 +83,24 @@ __aicore__ inline void RegbaseTwoPassPerfImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR
76 op.Process();83 op.Process();
77}84}
78 85 
86+template <typename T, typename U>
87+__aicore__ inline void RegbaseSplitRImpl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean, GM_ADDR rstd,
88+ GM_ADDR workspace, GM_ADDR tiling)
89+{
90+ if (workspace == nullptr) {
91+ return;
92+ }
93+ GM_ADDR userWS = GetUserWorkspace(workspace);
94+ if (userWS == nullptr) {
95+ return;
96+ }
97+ TPipe pipe;
98+ GET_TILING_DATA_WITH_STRUCT(LayerNormV4TilingDataSplitR, tilingData, tiling);
99+ LayerNormV4RegbaseSplitR<T, U, DTYPE_MEAN> op(&tilingData, &pipe);
100+ op.Init(x, gamma, beta, y, mean, rstd, userWS);
101+ op.Process();
102+}
103+ 
79static __aicore__ inline void layer_norm_impl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean,104static __aicore__ inline void layer_norm_impl(GM_ADDR x, GM_ADDR gamma, GM_ADDR beta, GM_ADDR y, GM_ADDR mean,
80 GM_ADDR rstd, GM_ADDR workspace, GM_ADDR tiling)105 GM_ADDR rstd, GM_ADDR workspace, GM_ADDR tiling)
81{106{
@@ -83,6 +108,32 @@ static __aicore__ inline void layer_norm_impl(GM_ADDR x, GM_ADDR gamma, GM_ADDR
83 return;108 return;
84 }109 }
85 KERNEL_TASK_TYPE_DEFAULT(KERNEL_TYPE_AIV_ONLY);110 KERNEL_TASK_TYPE_DEFAULT(KERNEL_TYPE_AIV_ONLY);
111+ // Hardware SyncAll requires MIX scheduling even when only AIV cores execute.
112+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT)) {
113+ KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_FLOAT_FLOAT, KERNEL_TYPE_MIX_AIV_1_0);
114+ RegbaseSplitRImpl<float, float>(x, gamma, beta, y, mean, rstd, workspace, tiling);
115+ return;
116+ }
117+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_HALF_FLOAT)) {
118+ KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_HALF_FLOAT, KERNEL_TYPE_MIX_AIV_1_0);
119+ RegbaseSplitRImpl<half, float>(x, gamma, beta, y, mean, rstd, workspace, tiling);
120+ return;
121+ }
122+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_HALF_HALF)) {
123+ KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_HALF_HALF, KERNEL_TYPE_MIX_AIV_1_0);
124+ RegbaseSplitRImpl<half, half>(x, gamma, beta, y, mean, rstd, workspace, tiling);
125+ return;
126+ }
127+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_BF16_FLOAT)) {
128+ KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_BF16_FLOAT, KERNEL_TYPE_MIX_AIV_1_0);
129+ RegbaseSplitRImpl<bfloat16_t, float>(x, gamma, beta, y, mean, rstd, workspace, tiling);
130+ return;
131+ }
132+ if (TILING_KEY_IS(LNV4_REGBASE_SPLIT_R_BF16_BF16)) {
133+ KERNEL_TASK_TYPE(LNV4_REGBASE_SPLIT_R_BF16_BF16, KERNEL_TYPE_MIX_AIV_1_0);
134+ RegbaseSplitRImpl<bfloat16_t, bfloat16_t>(x, gamma, beta, y, mean, rstd, workspace, tiling);
135+ return;
136+ }
86 if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) {137 if (TILING_KEY_IS(LNV4_REGBASE_TWO_PASS_FLOAT_FLOAT)) {
87 RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(x, gamma, beta, y, mean, rstd, tiling);138 RegbaseTwoPassImpl<float, float, DTYPE_MEAN>(x, gamma, beta, y, mean, rstd, tiling);
88 return;139 return;
@@ -21,3 +21,54 @@ LayerNormV4_float16_none_5,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), None, None)
21LayerNormV4_float16_none_6,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),()21LayerNormV4_float16_none_6,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),()
22LayerNormV4_float16_none_7,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), None, (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), None, (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),()22LayerNormV4_float16_none_7,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), None, (20000,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), None, (20000,))","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),()
23LayerNormV4_float16_none_8,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), None)","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),()23LayerNormV4_float16_none_8,UNKNOWN,layer_norm_v4,"((1, 20000), (1,), (20000,), None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 20000), (1,), (20000,), None)","('ND', 'ND', 'ND', 'ND')","((1, 20000), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,,,0,,(),()
24+LayerNormV4_fallback_float32_float32_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),()
25+LayerNormV4_split_r_float32_float32_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
26+LayerNormV4_split_r_float32_float32_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
27+LayerNormV4_split_r_float32_float32_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
28+LayerNormV4_fallback_float16_float16_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),()
29+LayerNormV4_split_r_float16_float16_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
30+LayerNormV4_split_r_float16_float16_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
31+LayerNormV4_split_r_float16_float16_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
32+LayerNormV4_fallback_float16_float32_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),()
33+LayerNormV4_split_r_float16_float32_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
34+LayerNormV4_split_r_float16_float32_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
35+LayerNormV4_split_r_float16_float32_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('float16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
36+LayerNormV4_fallback_bfloat16_bfloat16_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),()
37+LayerNormV4_split_r_bfloat16_bfloat16_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
38+LayerNormV4_split_r_bfloat16_bfloat16_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
39+LayerNormV4_split_r_bfloat16_bfloat16_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
40+LayerNormV4_fallback_bfloat16_float32_1_32768,UNKNOWN,layer_norm_v4,"((1, 32768), (1,), (32768,), (32768,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 32768), (1,), (32768,), (32768,))","('ND', 'ND', 'ND', 'ND')","((1, 32768), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,阈值下回退模板场景,,0,,(),()
41+LayerNormV4_split_r_bfloat16_float32_2_65537,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
42+LayerNormV4_split_r_bfloat16_float32_3_262145,UNKNOWN,layer_norm_v4,"((3, 262145), (1,), (262145,), (262145,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((3, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((3, 262145), (3, 1), (3, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
43+LayerNormV4_split_r_bfloat16_float32_1_1048576,UNKNOWN,layer_norm_v4,"((1, 1048576), (1,), (1048576,), (1048576,))","('bfloat16', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 1048576), (1,), (1048576,), (1048576,))","('ND', 'ND', 'ND', 'ND')","((1, 1048576), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,reduce轴多核:小A大R及非对齐尾段,,0,,(),()
44+LayerNormV4_split_r_float32_zero,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((0, 0), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
45+LayerNormV4_split_r_float32_constant,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((7, 7), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
46+LayerNormV4_split_r_float32_large_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1002), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
47+LayerNormV4_split_r_float16_zero,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((0, 0), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
48+LayerNormV4_split_r_float16_constant,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((7, 7), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
49+LayerNormV4_split_r_float16_large_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1002), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
50+LayerNormV4_split_r_bfloat16_zero,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((0, 0), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
51+LayerNormV4_split_r_bfloat16_constant,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((7, 7), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
52+LayerNormV4_split_r_bfloat16_large_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1002), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR数值边界:常数或大偏置小方差,,0,,(),()
53+LayerNormV4_split_r_float32_no_gamma,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), None, (65537,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), None, (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR可选参数缺省,,0,,(),()
54+LayerNormV4_split_r_float32_no_beta,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), None)","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR可选参数缺省,,0,,(),()
55+LayerNormV4_split_r_float32_no_params,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), None, None)","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), None, None)","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR可选参数缺省,,0,,(),()
56+LayerNormV4_split_r_float32_threshold_65535,UNKNOWN,layer_norm_v4,"((1, 65535), (1,), (65535,), (65535,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 65535), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65535), (1,), (65535,), (65535,))","('ND', 'ND', 'ND', 'ND')","((1, 65535), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR准入阈值边界,,0,,(),()
57+LayerNormV4_split_r_float32_threshold_65536,UNKNOWN,layer_norm_v4,"((1, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR准入阈值边界,,0,,(),()
58+LayerNormV4_split_r_asweep_float16_8_65536,UNKNOWN,layer_norm_v4,"((8, 65536), (1,), (65536,), (65536,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
59+LayerNormV4_split_r_asweep_float16_8_262145,UNKNOWN,layer_norm_v4,"((8, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
60+LayerNormV4_split_r_asweep_float16_16_65536,UNKNOWN,layer_norm_v4,"((16, 65536), (1,), (65536,), (65536,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
61+LayerNormV4_split_r_asweep_float16_16_262145,UNKNOWN,layer_norm_v4,"((16, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
62+LayerNormV4_split_r_asweep_float16_32_65536,UNKNOWN,layer_norm_v4,"((32, 65536), (1,), (65536,), (65536,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
63+LayerNormV4_split_r_asweep_float16_32_262145,UNKNOWN,layer_norm_v4,"((32, 262145), (1,), (262145,), (262145,))","('float16', 'int32', 'float16', 'float16')","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('float16', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
64+LayerNormV4_split_r_asweep_float32_8_65536,UNKNOWN,layer_norm_v4,"((8, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((8, 65536), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
65+LayerNormV4_split_r_asweep_float32_8_262145,UNKNOWN,layer_norm_v4,"((8, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((8, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((8, 262145), (8, 1), (8, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
66+LayerNormV4_split_r_asweep_float32_16_65536,UNKNOWN,layer_norm_v4,"((16, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((16, 65536), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
67+LayerNormV4_split_r_asweep_float32_16_262145,UNKNOWN,layer_norm_v4,"((16, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((16, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((16, 262145), (16, 1), (16, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
68+LayerNormV4_split_r_asweep_float32_32_65536,UNKNOWN,layer_norm_v4,"((32, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((32, 65536), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
69+LayerNormV4_split_r_asweep_float32_32_262145,UNKNOWN,layer_norm_v4,"((32, 262145), (1,), (262145,), (262145,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((32, 262145), (1,), (262145,), (262145,))","('ND', 'ND', 'ND', 'ND')","((32, 262145), (32, 1), (32, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1, 1), (-1, 1), (-1, 1), (-1, 1))","((0.004, 0.004),)",0.0001,(),(),TRUE,小A准入边界性能扫描,,0,,(),()
70+LayerNormV4_split_r_bfloat16_representable_offset,UNKNOWN,layer_norm_v4,"((2, 65537), (1,), (65537,), (65537,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((2, 65537), (1,), (65537,), (65537,))","('ND', 'ND', 'ND', 'ND')","((2, 65537), (2, 1), (2, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((1000, 1016), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,BF16在1000附近ULP约4,使用可表示扰动验证小方差,,0,,(),()
71+LayerNormV4_split_r_float32_finite_variance_65536,UNKNOWN,layer_norm_v4,"((1, 65536), (1,), (65536,), (65536,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),()
72+LayerNormV4_split_r_float32_finite_variance_81921,UNKNOWN,layer_norm_v4,"((1, 81921), (1,), (81921,), (81921,))","('float32', 'int32', 'float32', 'float32')","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('float32', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 81921), (1,), (81921,), (81921,))","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),()
73+LayerNormV4_split_r_bfloat16_finite_variance_65536,UNKNOWN,layer_norm_v4,"((1, 65536), (1,), (65536,), (65536,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 65536), (1,), (65536,), (65536,))","('ND', 'ND', 'ND', 'ND')","((1, 65536), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),()
74+LayerNormV4_split_r_bfloat16_finite_variance_81921,UNKNOWN,layer_norm_v4,"((1, 81921), (1,), (81921,), (81921,))","('bfloat16', 'int32', 'bfloat16', 'bfloat16')","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('bfloat16', 'float32', 'float32')","('ND', 'ND', 'ND')","((1, 81921), (1,), (81921,), (81921,))","('ND', 'ND', 'ND', 'ND')","((1, 81921), (1, 1), (1, 1))","('ND', 'ND', 'ND')",{'epsilon': 1e-05},"((-1152921504606846976, 1152921504606846976), (-1, 1), (1, 1), (0, 0))","((0.004, 0.004),)",0.0001,(),(),TRUE,SplitR有限大方差:先缩放count,避免variance乘分段长度溢出;覆盖4/5分段,,0,,(),()
@@ -14,6 +14,7 @@
14 */14 */
15 15 
16#include <iostream>16#include <iostream>
17+#include <cstring>
17#include <fstream>18#include <fstream>
18#include <vector>19#include <vector>
19#include <gtest/gtest.h>20#include <gtest/gtest.h>
@@ -25,6 +26,8 @@
25#include "kernel_run_context_facker.h"26#include "kernel_run_context_facker.h"
26#include "test_cube_util.h"27#include "test_cube_util.h"
27#include "base/err_mgr.h"28#include "base/err_mgr.h"
29+#include "tiling/platform/platform_ascendc.h"
30+#include "op_common/op_host/util/platform_util.h"
28 31 
29using namespace ut_util;32using namespace ut_util;
30using namespace std;33using namespace std;
@@ -2011,3 +2014,159 @@ static const auto _inst_LAYER_NORM_V4 = []() {
2011 testing::UnitTest::GetInstance()->listeners().Append(new LAYER_NORM_V4NoEz9999Check());2014 testing::UnitTest::GetInstance()->listeners().Append(new LAYER_NORM_V4NoEz9999Check());
2012 return 0;2015 return 0;
2013}();2016}();
2017+ 
2018+// Prevent regressions in SplitR dispatch and fallback at the shape boundaries.
2019+static void CheckLayerNormV4SplitR(int64_t a, int64_t r, int64_t expectedSplits)
2020+{
2021+ gert::StorageShape x_shape = {{a, r}, {a, r}};
2022+ gert::StorageShape normalized_shape = {{1}, {1}};
2023+ gert::StorageShape gamma_shape = {{r}, {r}};
2024+ gert::StorageShape beta_shape = {{r}, {r}};
2025+ gert::StorageShape y_shape = {{a, r}, {a, r}};
2026+ gert::StorageShape mean_shape = {{a}, {a}};
2027+ gert::StorageShape rstd_shape = {{a}, {a}};
2028+ float eps = 1e-5;
2029+ 
2030+ string compile_info_string = R"({
2031+ "hardware_info": {"BT_SIZE": 0, "load3d_constraints": "1",
2032+ "Intrinsic_fix_pipe_l0c2out": false,
2033+ "Intrinsic_data_move_l12ub": true,
2034+ "Intrinsic_data_move_l0c2ub": true,
2035+ "Intrinsic_data_move_out2l1_nd2nz": false,
2036+ "UB_SIZE": 245760, "L2_SIZE": 33554432, "L1_SIZE": 524288,
2037+ "L0A_SIZE": 65536, "L0B_SIZE": 65536, "L0C_SIZE": 131072,
2038+ "CORE_NUM": 64, "socVersion": "Ascend950"}
2039+ })";
2040+ map<string, string> soc_infos;
2041+ map<string, string> aicore_spec;
2042+ map<string, string> intrinsics;
2043+ map<string, string> soc_version;
2044+ map<string, string> npuarchs = {{"NpuArch", "3510"}};
2045+ GetPlatFormInfos(compile_info_string.c_str(), soc_infos, aicore_spec, intrinsics, soc_version);
2046+ 
2047+ fe::PlatFormInfos platform_info;
2048+ platform_info.Init();
2049+ LayerNormV4CompileInfo compile_info;
2050+ 
2051+ std::string op_type("LayerNormV4");
2052+ ASSERT_NE(gert::OpImplRegistry::GetInstance().GetOpImpl(op_type.c_str()), nullptr);
2053+ auto tiling_func = gert::OpImplRegistry::GetInstance().GetOpImpl(op_type.c_str())->tiling;
2054+ auto tiling_parse_func = gert::OpImplRegistry::GetInstance().GetOpImpl(op_type.c_str())->tiling_parse;
2055+ 
2056+ auto kernel_holder = gert::KernelRunContextFaker()
2057+ .KernelIONum(2, 1)
2058+ .Inputs({const_cast<char*>(compile_info_string.c_str()),
2059+ reinterpret_cast<void*>(&platform_info)})
2060+ .Outputs({&compile_info})
2061+ .Build();
2062+ 
2063+ ASSERT_TRUE(kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->Init());
2064+ kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("SoCInfo", soc_infos);
2065+ kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("AICoreSpec", aicore_spec);
2066+ kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetCoreNumByCoreType("AICore");
2067+ kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("AICoreintrinsicDtypeMap",
2068+ intrinsics);
2069+ kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("version", soc_version);
2070+ kernel_holder.GetContext<gert::TilingParseContext>()->GetPlatformInfo()->SetPlatformRes("version", npuarchs);
2071+ 
2072+ ASSERT_EQ(tiling_parse_func(kernel_holder.GetContext<gert::KernelContext>()), ge::GRAPH_SUCCESS);
2073+ 
2074+ auto param = gert::TilingData::CreateCap(4096);
2075+ auto workspace_size_holer = gert::ContinuousVector::Create<size_t>(4096);
2076+ auto ws_size = reinterpret_cast<gert::ContinuousVector*>(workspace_size_holer.get());
2077+ ASSERT_NE(param, nullptr);
2078+ auto holder = gert::TilingContextFaker()
2079+ .SetOpType(op_type)
2080+ .NodeIoNum(4, 3)
2081+ .IrInstanceNum({1, 1, 1, 1})
2082+ .InputShapes({&x_shape, &normalized_shape, &gamma_shape, &beta_shape})
2083+ .OutputShapes({&y_shape, &mean_shape, &rstd_shape})
2084+ .CompileInfo(&compile_info)
2085+ .PlatformInfo(reinterpret_cast<char*>(&platform_info))
2086+ .NodeInputTd(0, ge::DT_FLOAT16, ge::FORMAT_ND, ge::FORMAT_ND)
2087+ .NodeInputTd(1, ge::DT_INT32, ge::FORMAT_ND, ge::FORMAT_ND)
2088+ .NodeInputTd(2, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND)
2089+ .NodeInputTd(3, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND)
2090+ .NodeOutputTd(0, ge::DT_FLOAT16, ge::FORMAT_ND, ge::FORMAT_ND)
2091+ .NodeOutputTd(1, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND)
2092+ .NodeOutputTd(2, ge::DT_FLOAT, ge::FORMAT_ND, ge::FORMAT_ND)
2093+ .NodeAttrs({{"epsilon", Ops::NN::AnyValue::CreateFrom<float>(1e-05)}})
2094+ .TilingData(param.get())
2095+ .Workspace(ws_size)
2096+ .Build();
2097+ 
2098+ gert::TilingContext* tiling_context = holder.GetContext<gert::TilingContext>();
2099+ ASSERT_NE(tiling_context->GetPlatformInfo(), nullptr);
2100+ holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("SoCInfo", soc_infos);
2101+ holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("AICoreSpec", aicore_spec);
2102+ holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetCoreNumByCoreType("AICore");
2103+ holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("AICoreintrinsicDtypeMap", intrinsics);
atomgit-bot
atomgit-botatomgit-bot23 days ago

🟡 Medium Priority

变更行(test_layer_norm_v4_tiling.cpp:2100 与 2130-2133)只在 tiling_parse 上下文设置了 SetPlatformRes("version", soc_version),tiling 上下文则完全没有设置 "version"(既没有 soc_version 也没有 NpuArch)。

受影响契约:norm/layer_norm_v4/op_host/layer_norm_v4_split_r_tiling.cpp:37 的准入条件为 platform_ascendc::PlatformAscendC(info).GetCurNpuArch() != NpuArch::DAV_3510 || context_->GetDeterministicLevel() != 0 → return false,即 SplitR 必须在 GetCurNpuArch()==DAV_3510 时才会被选中;而 layer_norm_v4_tiling.cpp:62 的 isRegBase 同样依赖平台 arch 信息。

失败模式:CheckLayerNormV4SplitR(...) 三个 expectSplit=true 的用例(split_r_long_row / split_r_multiple_rows_and_tail / split_r_threshold_boundary)会走到 ASSERT_EQ(tiling_key, 810)(第 2138 行),但平台信息里没有 DAV_3510,SplitR 的 IsCapable 恒为 false,tiling_key 不会是 810(甚至可能因 isRegBase=false 回退到非 regbase 模板而让 2135 行的 GRAPH_SUCCESS 断言失败),新增的 6 个用例无法验证任何准入/回退行为。

对照证据:本工程所有 arch35 用例都在 tiling_parse 与 tiling 两个上下文同时设置 npuarchs = {{"NpuArch","3510"}}(同文件 454/483/518 行的 regbase 422 用例,以及 norm/renorm/tests/ut/op_host/arch35/test_renorm_tiling.cpp:72/106/139),新增 helper 两者都缺。

修复方向:在 helper 中声明 map<string,string> npuarchs = {{"NpuArch","3510"}};,并在 tiling_parse 上下文(第 2100 行之后)与 tiling 上下文(第 2130 行之前)都追加 SetPlatformRes("version", npuarchs),与既有 arch35 用例保持一致。

建议:在 helper 中增加 map<string,string> npuarchs = {{"NpuArch","3510"}};,并在 tiling_parse 上下文和 tiling 上下文都调用 SetPlatformRes("version", npuarchs),否则 SplitR 的 DAV_3510 准入条件不可能满足,expectSplit=true 的断言必然失败。

likedislike
Incorrect?
pipiman
pipiman
22 days ago Comments:
likedislike
2104+ holder.GetContext<gert::TilingContext>()->GetPlatformInfo()->SetPlatformRes("version", npuarchs);
2105+ 
2106+ ASSERT_EQ(tiling_func(tiling_context), ge::GRAPH_SUCCESS);
2107+ auto tiling_key = tiling_context->GetTilingKey();
2108+ if (expectedSplits > 0) {
2109+ ASSERT_EQ(tiling_key, 810);
2110+ EXPECT_EQ(tiling_context->GetBlockDim(), a * expectedSplits);
2111+ // SplitR publishes separate 32B mean and variance rows per core.
2112+ // Include the system SyncAll workspace.
2113+ auto platform = platform_ascendc::PlatformAscendC(tiling_context->GetPlatformInfo());
2114+ auto* workspace = tiling_context->GetWorkspaceSizes(1);
2115+ ASSERT_NE(workspace, nullptr);
2116+ EXPECT_EQ(workspace[0], platform.GetLibApiWorkSpaceSize() +
2117+ 2 * a * expectedSplits * Ops::Base::GetUbBlockSize(tiling_context));
2118+ } else {
2119+ EXPECT_NE(tiling_key, 810);
2120+ }
2121+ auto tilingData = tiling_context->GetRawTilingData();
2122+ ASSERT_NE(tilingData, nullptr);
2123+ if (expectedSplits > 0) {
2124+ // The common Welford/SplitR ABI starts with fourteen int64_t fields;
2125+ // read only this prefix, without depending on the epsilon/tail padding.
2126+ int64_t fields[14] = {};
2127+ ASSERT_GE(tilingData->GetDataSize(), sizeof(fields));
2128+ std::memcpy(fields, tilingData->GetData(), sizeof(fields));
2129+ constexpr size_t MAIN_BLOCK_FACTOR_FIELD = 5;
2130+ EXPECT_EQ(fields[MAIN_BLOCK_FACTOR_FIELD], 1); // One row per SplitR core.
2131+ constexpr size_t TILE_LENGTH_FIELD = 7;
2132+ constexpr size_t WELFORD_TEMP_FIELD = 8;
2133+ constexpr size_t API_TEMP_FIELD = 13;
2134+ const int64_t tileLength = fields[TILE_LENGTH_FIELD];
2135+ ASSERT_GT(tileLength, 0);
2136+ ASSERT_GT(compile_info.vectorLength, 0);
2137+ // FP16 x/y and FP32 gamma/beta, all double buffered; two double
2138+ // buffered statistics queues contain 256 floats each.
2139+ const int64_t ioBuffers = 24 * tileLength + 4 * 256 * sizeof(float);
2140+ const int64_t mergeBuffers = 2 * compile_info.vectorLength;
2141+ EXPECT_LE(ioBuffers + fields[WELFORD_TEMP_FIELD] + fields[API_TEMP_FIELD] + mergeBuffers,
2142+ compile_info.ubSizePlatForm);
2143+ }
2144+}
2145+ 
2146+TEST_F(LayerNormV4Tiling, split_r_long_row) { CheckLayerNormV4SplitR(1, 1048576, 16); }
2147+ 
2148+TEST_F(LayerNormV4Tiling, split_r_multiple_rows_and_tail) { CheckLayerNormV4SplitR(3, 65537, 4); }
2149+ 
2150+TEST_F(LayerNormV4Tiling, split_r_short_row_fallback) { CheckLayerNormV4SplitR(1, 65535, 0); }
2151+ 
2152+TEST_F(LayerNormV4Tiling, split_r_sufficient_rows_fallback) { CheckLayerNormV4SplitR(64, 65537, 0); }
2153+ 
2154+TEST_F(LayerNormV4Tiling, split_r_threshold_boundary) { CheckLayerNormV4SplitR(1, 65536, 4); }
2155+ 
2156+TEST_F(LayerNormV4Tiling, split_r_max_admitted_rows) { CheckLayerNormV4SplitR(16, 65536, 4); }
2157+ 
2158+TEST_F(LayerNormV4Tiling, split_r_above_max_admitted_rows) { CheckLayerNormV4SplitR(17, 65536, 0); }
2159+ 
2160+TEST_F(LayerNormV4Tiling, split_r_nearly_saturated_rows_fallback) { CheckLayerNormV4SplitR(32, 65536, 0); }
2161+ 
2162+// Non-power-of-two groups exercise compact masked reductions; a tail
2163+// element also makes the per-core counts unequal in the finalization input.
2164+TEST_F(LayerNormV4Tiling, split_r_five_parts_with_tail) { CheckLayerNormV4SplitR(1, 81921, 5); }
2165+ 
2166+TEST_F(LayerNormV4Tiling, split_r_six_parts_with_tail) { CheckLayerNormV4SplitR(1, 98305, 6); }
2167+ 
2168+TEST_F(LayerNormV4Tiling, split_r_seven_parts_core_limited) { CheckLayerNormV4SplitR(9, 1048577, 7); }
2169+ 
2170+TEST_F(LayerNormV4Tiling, split_r_nine_parts_with_tail) { CheckLayerNormV4SplitR(1, 147457, 9); }
2171+ 
2172+TEST_F(LayerNormV4Tiling, split_r_fifteen_parts_with_tail) { CheckLayerNormV4SplitR(1, 245761, 15); }
@@ -94,6 +94,43 @@ inline void InitTilingData(uint8_t* tiling, LayerNormV4TilingDataWelford* const_
94}94}
95#endif95#endif
96 96 
97+#pragma pack(1)
98+struct LayerNormV4TilingDataSplitR {
99+ int64_t M = 0;
100+ int64_t N = 0;
101+ int64_t rAlign = 0;
102+ int64_t numBlocks = 0;
103+ int64_t mainBlockCount = 0;
104+ int64_t mainBlockFactor = 0;
105+ int64_t tailBlockFactor = 0;
106+ int64_t tileLength = 0;
107+ int64_t welfordTempSize = 0;
108+ int64_t welfordUpdateTimes = 0;
109+ int64_t welfordUpdateTail = 0;
110+ int64_t nullptrGamma = 0;
111+ int64_t nullptrBeta = 0;
112+ int64_t apiTempBufferSize = 0;
113+ float epsilon = 0;
114+ int32_t pad_ = 0; // Match the host tiling serialization alignment before splitR.
115+ int64_t splitR = 0;
116+};
117+#pragma pack()
118+ 
119+#ifdef __NPU_TILING__
120+inline[aicore] void InitTilingData(const __gm__ uint8_t* tiling, LayerNormV4TilingDataSplitR* const_data)
121+{
122+ const __gm__ uint32_t* src = (const __gm__ uint32_t*)tiling;
123+ uint32_t* dst = (uint32_t*)const_data;
124+ for (auto i = 0; i < sizeof(LayerNormV4TilingDataSplitR) / 4; i++)
125+ *(dst + i) = *(src + i);
126+}
127+#else
128+inline void InitTilingData(uint8_t* tiling, LayerNormV4TilingDataSplitR* const_data)
129+{
130+ memcpy(const_data, tiling, sizeof(LayerNormV4TilingDataSplitR));
131+}
132+#endif
133+ 
97#define GET_TILING_DATA_WITH_STRUCT(tiling_struct, tiling_data, tiling_arg) \134#define GET_TILING_DATA_WITH_STRUCT(tiling_struct, tiling_data, tiling_arg) \
98 tiling_struct tiling_data; \135 tiling_struct tiling_data; \
99 InitTilingData(tiling_arg, &tiling_data)136 InitTilingData(tiling_arg, &tiling_data)