Merged
perf(in_training_reduce_v2): 优化短规约连续行计算 #13846
perf(in_training_reduce_v2): 优化短规约连续行计算 #13846
Merged
raoliang_sacAdd on 16 days ago
Total: 7 files changed+378-25
@@ -39,20 +39,21 @@ English | [简体中文](./README.md)
39| sum | Output | The result of summing `x` along the spatial dimensions. The instance dimension and the channel dimension are kept, and the size of each reduced dimension is 1. | FLOAT | NC1HWC0, NDC1HWC0, ND |39| sum | Output | The result of summing `x` along the spatial dimensions. The instance dimension and the channel dimension are kept, and the size of each reduced dimension is 1. | FLOAT | NC1HWC0, NDC1HWC0, ND |
40| square_sum | Output | The result of the square sum of `x` along the spatial dimensions. The shape is the same as `sum`. | FLOAT | NC1HWC0, NDC1HWC0, ND |40| square_sum | Output | The result of the square sum of `x` along the spatial dimensions. The shape is the same as `sum`. | FLOAT | NC1HWC0, NDC1HWC0, ND |
41 41 
42-### Product Differences
43- 
44-| Product | Static shape capability | Dynamic shape capability | Shape/rank and format mapping |
45-| --- | --- | --- | --- |
46-| <term>Ascend 950PR&950DT products</term> | `x` is NCHW, NCDHW, ND, NHWC, or NDHWC, and the two outputs are ND | Supports dynamic shapes of the preceding formats and ND dynamic rank. NHWC/NDHWC inputs keep the corresponding formats, NCHW/NCDHW/ND inputs use ND storage, and both outputs are ND | The NCHW input is `[N,C,H,W]` and the output is `[N,C,1,1]`; the NCDHW input is `[N,C,D,H,W]` and the output is `[N,C,1,1,1]`; the ND input has a rank of 2~8, the first two dimensions are N and C, and the remaining dimensions are reduced to 1; the NHWC input is `[N,H,W,C]` and the output is `[N,1,1,C]`; the NDHWC input is `[N,D,H,W,C]` and the output is `[N,1,1,1,C]` |
47-| <term>Atlas A3 products</term><br><term>Atlas A2 products</term><br><term>Atlas 200I/500 A2 inference products</term><br><term>Atlas inference products</term><br><term>Atlas training products</term> | The 4D NCHW/NHWC original formats are mapped to NC1HWC0, and the 5D NCDHW/NDHWC original formats are mapped to NDC1HWC0. The storage formats of the input and the output are consistent | The registered dynamic shape and dynamic rank paths of the corresponding models use NC1HWC0 or NDC1HWC0 storage, and the format mapping is consistent with the static shape | The original input is 4D or 5D. The logical output keeps N and C, and H, W or D, H, W is 1. The storage format is mapped by the graph engine |
48- 
49## Constraints42## Constraints
50 43 
51- The logical shape of NCHW/NHWC inputs is 4D, and the spatial dimensions are H and W. NCDHW/NDHWC inputs are 5D, and the spatial dimensions are D, H, and W.44- The logical shape of NCHW/NHWC inputs is 4D, and the spatial dimensions are H and W. NCDHW/NDHWC inputs are 5D, and the spatial dimensions are D, H, and W.
52-- For the <term>Ascend 950PR&950DT products</term>, the ND input rank is 2~8. dim0 and dim1 are the instance dimension and the channel dimension respectively, and dim2 and later dimensions are spatial dimensions.45+- <term>Ascend 950PR&950DT products</term>:
53-- The <term>Ascend 950PR&950DT products</term> require each dimension of `x` to be greater than 0. Empty tensors are not supported.46+ - For static shapes, `x` is NCHW, NCDHW, ND, NHWC, or NDHWC, and the two outputs are ND. Dynamic shapes of these formats and ND dynamic rank are supported. NHWC/NDHWC inputs keep the corresponding formats, NCHW/NCDHW/ND inputs use ND storage, and both outputs are ND.
54-- The <term>Ascend 950PR&950DT products</term> require the total number of input elements (the product of all dimension sizes) to not exceed `2^63-1`.47+ - The NCHW input is `[N,C,H,W]` and the output is `[N,C,1,1]`; the NCDHW input is `[N,C,D,H,W]` and the output is `[N,C,1,1,1]`.
55-- For the <term>Ascend 950PR&950DT products</term>, the channel row byte count `C*sizeof(x.dtype)` of NHWC/NDHWC inputs must not exceed `2^40-1`, that is, `C` of FLOAT16 must not exceed `2^39-1`, and `C` of FLOAT must not exceed `2^38-1`.48+ - The NHWC input is `[N,H,W,C]` and the output is `[N,1,1,C]`; the NDHWC input is `[N,D,H,W,C]` and the output is `[N,1,1,1,C]`.
49+ - The ND input rank is 2~8. dim0 and dim1 are the instance dimension and the channel dimension respectively, and dim2 and later dimensions are spatial dimensions reduced to 1.
50+ - Each dimension of `x` must be greater than 0. Empty tensors are not supported.
51+ - The total number of input elements (the product of all dimension sizes) must not exceed `2^63-1`.
52+ - The channel row byte count `C*sizeof(x.dtype)` of NHWC/NDHWC inputs must not exceed `2^40-1`, that is, `C` of FLOAT16 must not exceed `2^39-1`, and `C` of FLOAT must not exceed `2^38-1`.
53+- <term>Atlas A3 products</term>, <term>Atlas A2 products</term>, <term>Atlas 200I/500 A2 inference products</term>, <term>Atlas inference products</term>, and <term>Atlas training products</term>:
54+ - The 4D NCHW/NHWC original formats are mapped to NC1HWC0, and the 5D NCDHW/NDHWC original formats are mapped to NDC1HWC0. The storage formats of the input and the output are consistent.
55+ - The registered dynamic shape and dynamic rank paths of the corresponding models use NC1HWC0 or NDC1HWC0 storage, and the format mapping is consistent with the static shape.
56+ - The original input is 4D or 5D. The logical output keeps N and C, and H, W or D, H, W is 1. The storage format is mapped by the graph engine.
56 57 
57## Invocation Description58## Invocation Description
58 59 
@@ -113,6 +113,7 @@ protected:
113 ge::graphStatus PostTiling() override;113 ge::graphStatus PostTiling() override;
114 114 
115private:115private:
116+ bool DoPackedRowTiling(uint64_t elemSize);
116 // sub-R 分块 tiling(R 超单次 UB 容量)117 // sub-R 分块 tiling(R 超单次 UB 容量)
117 bool DoSubRTiling(uint64_t rAlign, uint64_t binAddQuotient, int64_t elemSize);118 bool DoSubRTiling(uint64_t rAlign, uint64_t binAddQuotient, int64_t elemSize);
118 // sub-R 路径 Kernel 侧 UB 精确占用(须与 op_kernel/arch35 的 InitSubR() 逐项一致)119 // sub-R 路径 Kernel 侧 UB 精确占用(须与 op_kernel/arch35 的 InitSubR() 逐项一致)
@@ -64,6 +64,13 @@ bool INTrainingReduceV2ARFullReduceTiling::IsCapable()
64 if (dataType == ge::DT_FLOAT16) {64 if (dataType == ge::DT_FLOAT16) {
65 elemSize = FP16_BYTE;65 elemSize = FP16_BYTE;
66 }66 }
67+ if (r <= vlfp32 && DoPackedRowTiling(static_cast<uint64_t>(elemSize))) {
68+ return true;
69+ }
70+ if (r == 1) {
71+ OP_LOGE(GetNodeName(), "R=1 input/output buffers exceed UB capacity");
72+ return false;
73+ }
67 74 
68 const uint64_t rValue = static_cast<uint64_t>(r);75 const uint64_t rValue = static_cast<uint64_t>(r);
69 const uint64_t elemSizeValue = static_cast<uint64_t>(elemSize);76 const uint64_t elemSizeValue = static_cast<uint64_t>(elemSize);
@@ -171,6 +178,45 @@ bool INTrainingReduceV2ARFullReduceTiling::IsCapable()
171 return true;178 return true;
172}179}
173 180 
181+// Pack consecutive N*C rows without per-row DMA padding. For short reductions,
182+// each vector lane owns one output row; R=1 keeps the elementwise fast path.
183+bool INTrainingReduceV2ARFullReduceTiling::DoPackedRowTiling(uint64_t elemSize)
184+{
185+ const uint64_t usable = aicoreParams_.ubSize - RESERVE_FOR_ALIGN;
186+ const uint64_t vectorElements = static_cast<uint64_t>(vlfp32);
187+ const uint64_t rowBytes = static_cast<uint64_t>(r) * elemSize;
188+ const uint64_t bytesPerRow = DOUBLE_BUFFER_COUNT * (rowBytes + REDUCE_OUTPUT_COUNT * sizeof(float));
189+ const uint64_t maxTileRows = usable / bytesPerRow / vectorElements * vectorElements;
190+ // Small mocked UB capacities can still use the original full-AR path.
191+ if (maxTileRows == 0) {
192+ return false;
193+ }
194+ const uint64_t balancedRows = Ops::Base::CeilAlign(Ops::Base::CeilDiv(totalRows_, aicoreParams_.blockDim),
195+ vectorElements);
196+ const uint64_t maxLoopRows = std::numeric_limits<uint16_t>::max() * vectorElements;
197+ const uint64_t maxCopyRows = std::numeric_limits<uint32_t>::max() / rowBytes / vectorElements * vectorElements;
198+ const uint64_t tileRows = std::min({maxTileRows, balancedRows, maxLoopRows, maxCopyRows});
199+ const uint64_t totalTiles = Ops::Base::CeilDiv(totalRows_, tileRows);
200+ const uint64_t tilesPerCore = Ops::Base::CeilDiv(totalTiles, aicoreParams_.blockDim);
201+ blockNum_ = Ops::Base::CeilDiv(totalTiles, tilesPerCore);
202+ 
203+ td_.numN = a1;
204+ td_.numC = a0;
205+ td_.numR = r;
206+ td_.rAlign = 0; // Packed rows use masked gathers, with no per-row padding.
207+ td_.cInner = static_cast<int64_t>(tileRows);
208+ td_.cOuter = static_cast<int64_t>(totalTiles);
209+ td_.cTail = static_cast<int64_t>(totalRows_ - (totalTiles - 1) * tileRows);
210+ td_.perCoreCnt = static_cast<int64_t>(tilesPerCore);
211+ td_.totalRows = static_cast<int64_t>(totalRows_);
212+ td_.totalElements = static_cast<int64_t>(totalElements_);
213+ td_.totalTiles = static_cast<int64_t>(totalTiles);
214+ // Full vector loads, including the last partial vector, stay inside UB.
215+ td_.inputBufferBytes = tileRows * rowBytes;
216+ td_.outputBufferBytes = tileRows * sizeof(float);
217+ return true;
218+}
219+ 
174// sub-R 路径 Kernel 侧 UB 精确占用。逐项对应 InitSubR():220// sub-R 路径 Kernel 侧 UB 精确占用。逐项对应 InitSubR():
175// inQueueX_ : CeilAlign(rFactor * elemSize, BLOCK_SIZE) * DOUBLE_BUFFER_NUM221// inQueueX_ : CeilAlign(rFactor * elemSize, BLOCK_SIZE) * DOUBLE_BUFFER_NUM
176// sum/sqPartialBuf_ : CeilAlign(slots, VL_FP32) * sizeof(float) * ACC_COMPONENTS,两块222// sum/sqPartialBuf_ : CeilAlign(slots, VL_FP32) * sizeof(float) * ACC_COMPONENTS,两块
@@ -53,6 +53,83 @@ __simd_callee__ inline void ReduceFullAR(RegTensor<float>& result, RegTensor<flo
53 Add(result, values, low, one);53 Add(result, values, low, one);
54}54}
55 55 
56+template <typename T_X>
57+__simd_vf__ inline void CalculateR1Vf(__ubuf__ T_X* xUb, __ubuf__ float* sumUb, __ubuf__ float* squareUb,
58+ uint32_t count, uint16_t loops)
59+{
60+ RegTensor<float> value;
61+ RegTensor<float> square;
62+ for (uint16_t i = 0; i < loops; ++i) {
63+ MaskReg valid = UpdateMask<float>(count);
64+ const uint32_t offset = static_cast<uint32_t>(i) * VL_FP32;
65+ if constexpr (IsSameType<T_X, float>::value) {
66+ AscendC::Reg::LoadAlign(value, xUb + offset);
67+ } else {
68+ RegTensor<T_X> input;
69+ AscendC::Reg::LoadAlign<T_X, LoadDist::DIST_UNPACK_B16>(input, xUb + offset);
70+ AscendC::Reg::Cast<float, T_X, castTraitB162B32>(value, input, valid);
71+ }
72+ Mul(square, value, value, valid);
73+ AscendC::Reg::StoreAlign(sumUb + offset, value, valid);
74+ AscendC::Reg::StoreAlign(squareUb + offset, square, valid);
75+ }
76+}
77+ 
78+template <typename T_X>
79+__simd_vf__ inline void CalculateShortRVf(__ubuf__ T_X* xUb, __ubuf__ float* sumUb, __ubuf__ float* squareUb,
80+ uint32_t rows, uint32_t reduceLen, uint16_t loops, float scale, float restore)
81+{
82+ RegTensor<uint32_t> rowOffsets;
83+ RegTensor<uint32_t> offsets;
84+ RegTensor<float> value;
85+ RegTensor<float> square;
86+ RegTensor<float> sumHi;
87+ RegTensor<float> sumLo;
88+ RegTensor<float> squareHi;
89+ RegTensor<float> squareLo;
90+ MaskReg full = CreateMask<float, MaskPattern::ALL>();
91+ AscendC::Reg::Arange((RegTensor<int32_t>&)rowOffsets, static_cast<int32_t>(0));
92+ Muls(rowOffsets, rowOffsets, reduceLen, full);
93+ for (uint16_t i = 0; i < loops; ++i) {
94+ const uint32_t rowOffset = static_cast<uint32_t>(i) * VL_FP32;
95+ uint32_t activeRows = rows;
96+ MaskReg valid = UpdateMask<float>(rows);
97+ activeRows -= rows;
98+ MaskReg validB16 = UpdateMask<half>(activeRows);
99+ Duplicate(sumHi, 0.0f, full);
100+ Duplicate(sumLo, 0.0f, full);
101+ Duplicate(squareHi, 0.0f, full);
102+ Duplicate(squareLo, 0.0f, full);
103+ __ubuf__ T_X* rowBase = xUb + rowOffset * reduceLen;
104+ for (uint16_t column = 0; column < reduceLen; ++column) {
105+ // ZEROING gives inactive lanes a safe zero index in the final tile.
106+ Adds(offsets, rowOffsets, static_cast<uint32_t>(column), valid);
107+ if constexpr (IsSameType<T_X, float>::value) {
108+ AscendC::Reg::Gather(value, rowBase, offsets, valid);
109+ } else {
110+ RegTensor<uint16_t> offsetsB16;
111+ RegTensor<T_X> gathered;
112+ RegTensor<T_X> unpacked;
113+ AscendC::Reg::Pack(offsetsB16, offsets);
114+ AscendC::Reg::Gather(gathered, rowBase, offsetsB16, validB16);
115+ AscendC::Reg::UnPack((RegTensor<uint32_t>&)unpacked, (RegTensor<uint16_t>&)gathered);
116+ AscendC::Reg::Cast<float, T_X, castTraitB162B32>(value, unpacked, valid);
117+ }
118+ Mul(square, value, value, valid);
119+ ScaleSumValue<T_X>(value, scale, valid);
120+ AccumulateWithError(sumHi, sumLo, value, valid);
121+ AccumulateWithError(squareHi, squareLo, square, valid);
122+ }
123+ ClearNonFiniteCorrection(sumLo, valid);
124+ ClearNonFiniteCorrection(squareLo, valid);
125+ Add(sumHi, sumHi, sumLo, valid);
126+ Add(squareHi, squareHi, squareLo, valid);
127+ ScaleSumValue<T_X>(sumHi, restore, valid);
128+ AscendC::Reg::StoreAlign(sumUb + rowOffset, sumHi, valid);
129+ AscendC::Reg::StoreAlign(squareUb + rowOffset, squareHi, valid);
130+ }
131+}
132+ 
56template <typename T_X>133template <typename T_X>
57class INTrainingReduceV2ARFullReduce {134class INTrainingReduceV2ARFullReduce {
58public:135public:
@@ -102,6 +179,9 @@ public:
102 pipe_.InitBuffer(inQueueX_, DOUBLE_BUFFER_NUM, inputBufferBytes_);179 pipe_.InitBuffer(inQueueX_, DOUBLE_BUFFER_NUM, inputBufferBytes_);
103 pipe_.InitBuffer(outQueueSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_);180 pipe_.InitBuffer(outQueueSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_);
104 pipe_.InitBuffer(outQueueSquareSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_);181 pipe_.InitBuffer(outQueueSquareSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_);
182+ if (rAlign_ == 0) {
183+ return;
184+ }
105 pipe_.InitBuffer(binaryAddBuf_, scratchBufferBytes_);185 pipe_.InitBuffer(binaryAddBuf_, scratchBufferBytes_);
106 pipe_.InitBuffer(squareBinaryAddBuf_, scratchBufferBytes_);186 pipe_.InitBuffer(squareBinaryAddBuf_, scratchBufferBytes_);
107 }187 }
@@ -123,6 +203,10 @@ public:
123 ProcessSubR();203 ProcessSubR();
124 return;204 return;
125 }205 }
206+ if (rAlign_ == 0) {
207+ ProcessPackedRows();
208+ return;
209+ }
126 const int64_t totalCnt = totalTiles_;210 const int64_t totalCnt = totalTiles_;
127 const int64_t startIndex = blockIdx_ * perCoreCnt_;211 const int64_t startIndex = blockIdx_ * perCoreCnt_;
128 if (startIndex >= totalCnt) {212 if (startIndex >= totalCnt) {
@@ -152,6 +236,46 @@ public:
152 }236 }
153 237 
154private:238private:
239+ __aicore__ inline void ProcessPackedRows()
240+ {
241+ const int64_t startIndex = blockIdx_ * perCoreCnt_;
242+ if (startIndex >= totalTiles_) {
243+ return;
244+ }
245+ const int64_t remaining = totalTiles_ - startIndex;
246+ const int64_t endIndex = startIndex + (perCoreCnt_ < remaining ? perCoreCnt_ : remaining);
247+ for (int64_t tile = startIndex; tile < endIndex; ++tile) {
248+ const int64_t offset = tile * cInner_;
249+ const uint32_t count = static_cast<uint32_t>(tile == totalTiles_ - 1 ? cTail_ : cInner_);
250+ // A single packed DMA replaces one padded DMA block per output row.
251+ const uint32_t inputCount = count * static_cast<uint32_t>(numR_);
252+ CopyInX(offset * numR_, 1, inputCount, inputCount);
253+ LocalTensor<T_X> xLocal = inQueueX_.DeQue<T_X>();
254+ LocalTensor<TSum> sumLocal = outQueueSum_.AllocTensor<TSum>();
255+ LocalTensor<TSum> squareSumLocal = outQueueSquareSum_.AllocTensor<TSum>();
256+ CalculatePackedRows(xLocal, sumLocal, squareSumLocal, count);
257+ inQueueX_.FreeTensor(xLocal);
258+ outQueueSum_.EnQue(sumLocal);
259+ outQueueSquareSum_.EnQue(squareSumLocal);
260+ CopyOutSumSquareSum(offset, count);
261+ }
262+ }
263+ 
264+ __aicore__ inline void CalculatePackedRows(const LocalTensor<T_X>& xLocal, LocalTensor<TSum>& sumLocal,
265+ LocalTensor<TSum>& squareSumLocal, uint32_t count)
266+ {
267+ __ubuf__ T_X* xUb = (__ubuf__ T_X*)xLocal.GetPhyAddr();
268+ __ubuf__ float* sumUb = (__ubuf__ float*)sumLocal.GetPhyAddr();
269+ __ubuf__ float* squareUb = (__ubuf__ float*)squareSumLocal.GetPhyAddr();
270+ const uint16_t loops = static_cast<uint16_t>((count + VL_FP32 - 1) / VL_FP32);
271+ if (numR_ == 1) {
272+ asc_vf_call<CalculateR1Vf<T_X>>(xUb, sumUb, squareUb, count, loops);
273+ } else {
274+ asc_vf_call<CalculateShortRVf<T_X>>(xUb, sumUb, squareUb, count, static_cast<uint32_t>(numR_), loops,
275+ sumScale_, sumRestore_);
276+ }
277+ }
278+ 
155 __aicore__ inline void CopyInX(int64_t offset, uint32_t cnt, uint32_t r, uint32_t rAlign)279 __aicore__ inline void CopyInX(int64_t offset, uint32_t cnt, uint32_t r, uint32_t rAlign)
156 {280 {
157 LocalTensor<T_X> xLocal = inQueueX_.AllocTensor<T_X>();281 LocalTensor<T_X> xLocal = inQueueX_.AllocTensor<T_X>();
@@ -22,10 +22,10 @@ struct INTrainingReduceV2ARFullReduceTilingData {
22 int64_t numN; // N 轴(a1)22 int64_t numN; // N 轴(a1)
23 int64_t numC; // C 轴(a0)23 int64_t numC; // C 轴(a0)
24 int64_t numR; // 规约轴长度 R = H*W(5D: D*H*W)24 int64_t numR; // 规约轴长度 R = H*W(5D: D*H*W)
25- uint64_t rAlign; // R 按 dtype 元素宽度 32B 对齐后的长度25+ uint64_t rAlign; // R 的 32B 对齐长度;0 表示短 R 连续打包路径
26- int64_t cInner; // 单 tile 内一次处理的 C 行数(sub-R 分块路径恒为 1)26+ int64_t cInner; // 单 tile 的 C 行数;打包路径为展平 N*C 后的行数;sub-R 恒为 1
27- int64_t cOuter; // C 轴切分的外层块数27+ int64_t cOuter; // C 轴块数;打包路径为展平 N*C 后的块数
28- int64_t cTail; // C 轴尾块长度28+ int64_t cTail; // C 轴尾块长度;打包路径为展平 N*C 后的尾块长度
29 uint64_t binaryAddQuotient; // 小于 rAlign 的最大 2 的幂(pairwise 折叠点)29 uint64_t binaryAddQuotient; // 小于 rAlign 的最大 2 的幂(pairwise 折叠点)
30 int64_t perCoreCnt; // 平均每个核处理的 tile 数30 int64_t perCoreCnt; // 平均每个核处理的 tile 数
31 // ---- sub-R 分块路径(R 超单次 UB 容量时启用)----31 // ---- sub-R 分块路径(R 超单次 UB 容量时启用)----
@@ -40,7 +40,7 @@ struct INTrainingReduceV2ARFullReduceTilingData {
40 uint64_t tailChunks; // 末组的 chunk 数 = numChunks - (numGroups-1)*chunksPerGroup40 uint64_t tailChunks; // 末组的 chunk 数 = numChunks - (numGroups-1)*chunksPerGroup
41 int64_t totalRows; // N*C,由 host 做溢出校验后下发41 int64_t totalRows; // N*C,由 host 做溢出校验后下发
42 int64_t totalElements; // N*C*R,由 host 做溢出校验后下发42 int64_t totalElements; // N*C*R,由 host 做溢出校验后下发
43- int64_t totalTiles; // full-load 路径的 N*cOuter;sub-R 路径等于 totalRows43+ int64_t totalTiles; // full-load: N*cOuter;打包路径: cOuter;sub-R: totalRows
44 uint64_t inputBufferBytes; // inQueueX 单 buffer 字节数44 uint64_t inputBufferBytes; // inQueueX 单 buffer 字节数
45 uint64_t outputBufferBytes; // sum/square_sum 各自单 buffer 字节数45 uint64_t outputBufferBytes; // sum/square_sum 各自单 buffer 字节数
46 uint64_t scratchBufferBytes; // full-load 路径每个 binary-add scratch 的字节数46 uint64_t scratchBufferBytes; // full-load 路径每个 binary-add scratch 的字节数
@@ -19,3 +19,11 @@ h31_smallc_float32_r63,UNKNOWN,in_training_reduce_v2,"((1, 1, 63, 7),)","('float
19h31_smallc_float32_r64,UNKNOWN,in_training_reduce_v2,"((1, 1, 64, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 64, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),()19h31_smallc_float32_r64,UNKNOWN,in_training_reduce_v2,"((1, 1, 64, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 64, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),()
20h31_smallc_float32_r65,UNKNOWN,in_training_reduce_v2,"((1, 1, 65, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 65, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),()20h31_smallc_float32_r65,UNKNOWN,in_training_reduce_v2,"((1, 1, 65, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 65, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),()
21h31_subr_float32_tail65,UNKNOWN,in_training_reduce_v2,"((1, 1, 131137),)","('float32',)","('ND',)","((1, 1, 1), (1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((1, 1, 131137),)","('ND',)","((1, 1, 1), (1, 1, 1))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),()21h31_subr_float32_tail65,UNKNOWN,in_training_reduce_v2,"((1, 1, 131137),)","('float32',)","('ND',)","((1, 1, 1), (1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((1, 1, 131137),)","('ND',)","((1, 1, 1), (1, 1, 1))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),()
22+r1_flat_rows_fp16,UNKNOWN,in_training_reduce_v2,"((131, 1),)","('float16',)","('ND',)","((131, 1), (131, 1))","('float32', 'float32')","('ND', 'ND')","((131, 1),)","('ND',)","((131, 1), (131, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),()
23+r1_flat_tail_fp32,UNKNOWN,in_training_reduce_v2,"((3, 65),)","('float32',)","('ND',)","((3, 65), (3, 65))","('float32', 'float32')","('ND', 'ND')","((3, 65),)","('ND',)","((3, 65), (3, 65))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),()
24+r1_nchw_tail_fp16,UNKNOWN,in_training_reduce_v2,"((3, 65, 1, 1),)","('float16',)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 1, 1),)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),()
25+r1_ncdhw_tail_fp32,UNKNOWN,in_training_reduce_v2,"((3, 65, 1, 1, 1),)","('float32',)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 1, 1, 1),)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),()
26+short_r7_rows131_fp16,UNKNOWN,in_training_reduce_v2,"((131, 1, 7),)","('float16',)","('ND',)","((131, 1, 1), (131, 1, 1))","('float32', 'float32')","('ND', 'ND')","((131, 1, 7),)","('ND',)","((131, 1, 1), (131, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),()
27+short_r9_tail_fp32,UNKNOWN,in_training_reduce_v2,"((3, 65, 3, 3),)","('float32',)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 3, 3),)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),()
28+short_r63_ncdhw_fp16,UNKNOWN,in_training_reduce_v2,"((3, 65, 1, 7, 9),)","('float16',)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 1, 7, 9),)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),()
29+short_r64_ndhwc_c1_fp32,UNKNOWN,in_training_reduce_v2,"((65, 4, 4, 4, 1),)","('float32',)","('NDHWC',)","((65, 1, 1, 1, 1), (65, 1, 1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((65, 4, 4, 4, 1),)","('NDHWC',)","((65, 1, 1, 1, 1), (65, 1, 1, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),()
@@ -70,7 +70,8 @@ static uint64_t RunTiling(gert::StorageShape& xShape, gert::StorageShape& outSha
70 INTrainingReduceV2ARFullReduceTilingData* tdOut = nullptr,70 INTrainingReduceV2ARFullReduceTilingData* tdOut = nullptr,
71 ge::DataType sumDtype = ge::DT_FLOAT, ge::DataType squareSumDtype = ge::DT_FLOAT,71 ge::DataType sumDtype = ge::DT_FLOAT, ge::DataType squareSumDtype = ge::DT_FLOAT,
72 uint64_t ubSize = 245760, INTrainingReduceV2ARAReduceTilingData* araOut = nullptr,72 uint64_t ubSize = 245760, INTrainingReduceV2ARAReduceTilingData* araOut = nullptr,
73- gert::StorageShape* squareSumShape = nullptr, ge::graphStatus* tilingStatus = nullptr)73+ gert::StorageShape* squareSumShape = nullptr, ge::graphStatus* tilingStatus = nullptr,
74+ uint32_t* blockDimOut = nullptr)
74{75{
75 std::map<std::string, std::string> socVersionInfos = {{"NpuArch", "3510"}};76 std::map<std::string, std::string> socVersionInfos = {{"NpuArch", "3510"}};
76 map<string, string> socInfos;77 map<string, string> socInfos;
@@ -153,6 +154,9 @@ static uint64_t RunTiling(gert::StorageShape& xShape, gert::StorageShape& outSha
153 if (status != ge::GRAPH_SUCCESS) {154 if (status != ge::GRAPH_SUCCESS) {
154 return UINT64_MAX;155 return UINT64_MAX;
155 }156 }
157+ if (blockDimOut != nullptr) {
158+ *blockDimOut = tilingContext->GetBlockDim();
159+ }
156 if (tdOut != nullptr) {160 if (tdOut != nullptr) {
157 auto raw = tilingContext->GetRawTilingData();161 auto raw = tilingContext->GetRawTilingData();
158 if (raw == nullptr || raw->GetDataSize() < sizeof(*tdOut)) {162 if (raw == nullptr || raw->GetDataSize() < sizeof(*tdOut)) {
@@ -218,6 +222,173 @@ static void ExpectSubRSelfConsistent(const INTrainingReduceV2ARFullReduceTilingD
218// ---------------------------------------------------------------------------222// ---------------------------------------------------------------------------
219// AR_FULL_REDUCE:4D NCHW 典型 shape [4,16,32,32] → TilingKey 200000223// AR_FULL_REDUCE:4D NCHW 典型 shape [4,16,32,32] → TilingKey 200000
220// ---------------------------------------------------------------------------224// ---------------------------------------------------------------------------
225+TEST_F(INTrainingReduceV2TilingTest, R1PacksIndependentRowsAcrossInstanceBoundaries)
226+{
227+ const std::vector<std::vector<int64_t>> shapes = {{1, 1}, {1, 63}, {1, 64}, {1, 65},
228+ {131, 1}, {120, 205}, {2144, 1962}};
229+ for (auto dtype : {ge::DT_FLOAT16, ge::DT_FLOAT}) {
230+ const uint64_t elemSize = dtype == ge::DT_FLOAT16 ? sizeof(uint16_t) : sizeof(float);
231+ for (const auto& dims : shapes) {
232+ SCOPED_TRACE(::testing::Message() << "N=" << dims[0] << " C=" << dims[1] << " dtype=" << dtype);
233+ gert::StorageShape input = {{dims[0], dims[1]}, {dims[0], dims[1]}};
234+ gert::StorageShape output = input;
235+ INTrainingReduceV2ARFullReduceTilingData td{};
236+ uint32_t blocks = 0;
237+ ASSERT_EQ(RunTiling(input, output, dtype, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, MOCK_UB_SIZE,
238+ nullptr, nullptr, nullptr, &blocks),
239+ EXPECTED_AR_TILING_KEY);
240+ ASSERT_EQ(td.numR, 1);
241+ ASSERT_EQ(td.totalRows, dims[0] * dims[1]);
242+ ASSERT_EQ(td.totalElements, td.totalRows);
243+ ASSERT_GT(td.cInner, 0);
244+ EXPECT_EQ(td.cInner % 64, 0);
245+ EXPECT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, td.totalRows);
246+ EXPECT_GT(td.cTail, 0);
247+ EXPECT_LE(td.cTail, td.cInner);
248+ EXPECT_GE(td.inputBufferBytes, static_cast<uint64_t>(td.cInner) * elemSize);
249+ EXPECT_GE(td.outputBufferBytes, static_cast<uint64_t>(td.cInner) * sizeof(float));
250+ EXPECT_EQ(td.scratchBufferBytes, 0U);
251+ EXPECT_EQ(td.partialBufferBytes, 0U);
252+ EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, MOCK_UB_SIZE);
253+ ASSERT_GT(blocks, 0U);
254+ EXPECT_LE(blocks, 64U);
255+ EXPECT_LT((blocks - 1) * td.perCoreCnt, td.totalTiles);
256+ EXPECT_GE(blocks * td.perCoreCnt, td.totalTiles);
257+ if (dims[0] == 131 && dims[1] == 1) {
258+ // N must no longer force one padded reduction and DMA per row.
259+ EXPECT_GT(td.cInner, td.numC);
260+ EXPECT_EQ(td.totalTiles, 3);
261+ }
262+ }
263+ }
264+}
265+ 
266+TEST_F(INTrainingReduceV2TilingTest, R1KeepsChannelFirstAndSingleChannelLayoutContracts)
267+{
268+ struct LayoutCase {
269+ std::vector<int64_t> dims;
270+ ge::Format format;
271+ };
272+ const std::vector<LayoutCase> cases = {
273+ {{3, 65, 1}, ge::FORMAT_ND}, {{3, 65, 1, 1}, ge::FORMAT_NCHW},
274+ {{3, 65, 1, 1, 1}, ge::FORMAT_NCDHW}, {{3, 65, 1, 1, 1, 1, 1, 1}, ge::FORMAT_ND},
275+ {{131, 1, 1, 1}, ge::FORMAT_NHWC}, {{131, 1, 1, 1, 1}, ge::FORMAT_NDHWC}};
276+ for (const auto& item : cases) {
277+ for (auto dtype : {ge::DT_FLOAT16, ge::DT_FLOAT}) {
278+ gert::Shape shape;
279+ for (int64_t dim : item.dims) {
280+ shape.AppendDim(dim);
281+ }
282+ gert::StorageShape input;
283+ input.MutableOriginShape() = shape;
284+ input.MutableStorageShape() = shape;
285+ gert::StorageShape output = input;
286+ INTrainingReduceV2ARFullReduceTilingData td{};
287+ ASSERT_EQ(RunTiling(input, output, dtype, item.format, &td), EXPECTED_AR_TILING_KEY);
288+ EXPECT_EQ(td.numR, 1);
289+ EXPECT_EQ(td.totalRows, item.dims[0] * item.dims[1]);
290+ EXPECT_EQ(td.totalTiles, td.cOuter);
291+ EXPECT_EQ(td.scratchBufferBytes, 0U);
292+ }
293+ }
294+}
295+ 
296+TEST_F(INTrainingReduceV2TilingTest, R1Preserves64BitOffsetsAndBoundsVectorLoopCount)
297+{
298+ for (uint64_t ubSize : {MOCK_UB_SIZE, uint64_t{1} << 28}) {
299+ gert::StorageShape input = {{131073, 65537}, {131073, 65537}};
300+ gert::StorageShape output = input;
301+ INTrainingReduceV2ARFullReduceTilingData td{};
302+ ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, ubSize),
303+ EXPECTED_AR_TILING_KEY);
304+ EXPECT_EQ(td.totalRows, 8590131201LL);
305+ EXPECT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, td.totalRows);
306+ EXPECT_GT((td.totalTiles - 1) * td.cInner, static_cast<int64_t>(UINT32_MAX));
307+ EXPECT_LE(td.cInner / 64, std::numeric_limits<uint16_t>::max());
308+ EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, ubSize);
309+ }
310+}
311+ 
312+TEST_F(INTrainingReduceV2TilingTest, R1HonorsMinimumUbBudget)
313+{
314+ gert::StorageShape input = {{120, 205}, {120, 205}};
315+ gert::StorageShape output = input;
316+ INTrainingReduceV2ARFullReduceTilingData td{};
317+ ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, 2048),
318+ EXPECTED_AR_TILING_KEY);
319+ EXPECT_EQ(td.cInner, 64);
320+ EXPECT_EQ(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, 2048U);
321+ EXPECT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, nullptr, ge::DT_FLOAT, ge::DT_FLOAT, 2047),
322+ UINT64_MAX);
323+}
324+ 
325+TEST_F(INTrainingReduceV2TilingTest, ShortRPacksRowsWithinUbAndDmaBounds)
326+{
327+ for (auto dtype : {ge::DT_FLOAT16, ge::DT_FLOAT}) {
328+ const uint64_t elemSize = dtype == ge::DT_FLOAT16 ? sizeof(uint16_t) : sizeof(float);
329+ for (int64_t reduceSize : {2, 3, 7, 8, 9, 16, 32, 63, 64}) {
330+ for (int64_t rows : {1, 63, 64, 65, 131, 951603}) {
331+ SCOPED_TRACE(::testing::Message() << "rows=" << rows << " R=" << reduceSize << " dtype=" << dtype);
332+ gert::StorageShape input = {{rows, 1, reduceSize}, {rows, 1, reduceSize}};
333+ gert::StorageShape output = {{rows, 1, 1}, {rows, 1, 1}};
334+ INTrainingReduceV2ARFullReduceTilingData td{};
335+ uint32_t blocks = 0;
336+ ASSERT_EQ(RunTiling(input, output, dtype, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, MOCK_UB_SIZE,
337+ nullptr, nullptr, nullptr, &blocks),
338+ EXPECTED_AR_TILING_KEY);
339+ ASSERT_EQ(td.rAlign, 0U);
340+ EXPECT_EQ(td.isSubRTiling, 0U);
341+ ASSERT_GT(td.cInner, 0);
342+ EXPECT_EQ(td.cInner % 64, 0);
343+ EXPECT_EQ(td.totalRows, rows);
344+ EXPECT_EQ(td.totalElements, rows * reduceSize);
345+ EXPECT_EQ(td.totalTiles, td.cOuter);
346+ EXPECT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, rows);
347+ EXPECT_GT(td.cTail, 0);
348+ EXPECT_LE(td.cTail, td.cInner);
349+ EXPECT_EQ(td.inputBufferBytes, td.cInner * reduceSize * elemSize);
350+ EXPECT_EQ(td.outputBufferBytes, td.cInner * sizeof(float));
351+ EXPECT_LE(td.inputBufferBytes, UINT32_MAX);
352+ EXPECT_EQ(td.scratchBufferBytes, 0U);
353+ EXPECT_EQ(td.partialBufferBytes, 0U);
354+ EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, MOCK_UB_SIZE);
355+ ASSERT_GT(blocks, 0U);
356+ EXPECT_LE(blocks, 64U);
357+ EXPECT_LT((blocks - 1) * td.perCoreCnt, td.totalTiles);
358+ EXPECT_GE(blocks * td.perCoreCnt, td.totalTiles);
359+ }
360+ }
361+ }
362+}
363+ 
364+TEST_F(INTrainingReduceV2TilingTest, ShortRRetainsFullArFallbackAndLargeRowOffsets)
365+{
366+ for (int64_t reduceSize : {2, 64, 65}) {
367+ gert::StorageShape input = {{65537, 65537, reduceSize}, {65537, 65537, reduceSize}};
368+ gert::StorageShape output = {{65537, 65537, 1}, {65537, 65537, 1}};
369+ INTrainingReduceV2ARFullReduceTilingData td{};
370+ ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td), EXPECTED_AR_TILING_KEY);
371+ EXPECT_EQ(td.totalRows, 4295098369LL);
372+ EXPECT_EQ(td.totalElements, td.totalRows * reduceSize);
373+ if (reduceSize <= 64) {
374+ EXPECT_EQ(td.rAlign, 0U);
375+ EXPECT_GT((td.totalTiles - 1) * td.cInner * reduceSize, static_cast<int64_t>(UINT32_MAX));
376+ } else {
377+ EXPECT_GT(td.rAlign, 0U);
378+ EXPECT_EQ(td.totalTiles, td.numN * td.cOuter);
379+ }
380+ }
381+ gert::StorageShape input = {{131, 1, 2}, {131, 1, 2}};
382+ gert::StorageShape output = {{131, 1, 1}, {131, 1, 1}};
383+ INTrainingReduceV2ARFullReduceTilingData td{};
384+ ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, 2048),
385+ EXPECTED_AR_TILING_KEY);
386+ // A packed 64-row tile needs 2560 bytes; the original full-AR route still fits.
387+ EXPECT_GT(td.rAlign, 0U);
388+ EXPECT_EQ(td.isSubRTiling, 0U);
389+ EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 2 * td.scratchBufferBytes, 2048U);
390+}
391+ 
221TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nchw_200000_001)392TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nchw_200000_001)
222{393{
223 gert::StorageShape x_shape = {{4, 16, 32, 32}, {4, 16, 32, 32}};394 gert::StorageShape x_shape = {{4, 16, 32, 32}, {4, 16, 32, 32}};
@@ -588,9 +759,7 @@ TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_sub_r_nc_over_uint32_
588// ---------------------------------------------------------------------------759// ---------------------------------------------------------------------------
589// N*C 溢出 uint32 且 R 很小时,full-load 路径同样必须被接受。760// N*C 溢出 uint32 且 R 很小时,full-load 路径同样必须被接受。
590// 本例 N=65536, C=65536(N*C=2^32 > UINT32_MAX)、R=4:761// 本例 N=65536, C=65536(N*C=2^32 > UINT32_MAX)、R=4:
591-// rAlign = CeilAlign(4*4, 32)/4 = 8762+// 短规约按连续输出行打包;tiling 和 GM 行偏移全程 64 位,无截断。
592-// cInner = (245760-512) / (8*4*2 + 4*2*2 + 32*2) = 245248/144 = 1703 >= 1
593-// 故走 full-load;tiling 全程 64 位,无截断。
594// ---------------------------------------------------------------------------763// ---------------------------------------------------------------------------
595TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nc_over_uint32_small_r_accepted_019)764TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nc_over_uint32_small_r_accepted_019)
596{765{
@@ -609,7 +778,9 @@ TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nc_over_uint32_small_
609 ASSERT_EQ(td.numN * td.numC, 4294967296LL);778 ASSERT_EQ(td.numN * td.numC, 4294967296LL);
610 ASSERT_EQ(td.totalRows, 4294967296LL);779 ASSERT_EQ(td.totalRows, 4294967296LL);
611 ASSERT_EQ(td.totalElements, 17179869184LL);780 ASSERT_EQ(td.totalElements, 17179869184LL);
612- ASSERT_EQ(td.totalTiles, td.numN * td.cOuter);781+ ASSERT_EQ(td.rAlign, 0U);
782+ ASSERT_EQ(td.totalTiles, td.cOuter);
783+ ASSERT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, td.totalRows);
613}784}
614 785 
615// ---------------------------------------------------------------------------786// ---------------------------------------------------------------------------
@@ -1024,10 +1195,12 @@ TEST_F(INTrainingReduceV2TilingTest, ArLastVectorLoadFitsInputAllocation)
1024 INTrainingReduceV2ARFullReduceTilingData td{};1195 INTrainingReduceV2ARFullReduceTilingData td{};
1025 ASSERT_EQ(RunTiling(xShape, outShape, dtype, ge::FORMAT_NCHW, &td), 200000U);1196 ASSERT_EQ(RunTiling(xShape, outShape, dtype, ge::FORMAT_NCHW, &td), 200000U);
1026 ASSERT_EQ(td.isSubRTiling, 0U);1197 ASSERT_EQ(td.isSubRTiling, 0U);
1027- const uint64_t finalReadEnd = ((td.cInner - 1U) * td.rAlign +1198+ const uint64_t finalReadEnd = td.rAlign == 0 ?
1028- (static_cast<uint64_t>(reduceSize) + kVectorElementsFp32 - 1U) /1199+ td.cInner * reduceSize * elemSize :
1029- kVectorElementsFp32 * kVectorElementsFp32) *1200+ ((td.cInner - 1U) * td.rAlign +
1030- elemSize;1201+ (static_cast<uint64_t>(reduceSize) + kVectorElementsFp32 - 1U) /
1202+ kVectorElementsFp32 * kVectorElementsFp32) *
1203+ elemSize;
1031 EXPECT_GE(td.inputBufferBytes, finalReadEnd);1204 EXPECT_GE(td.inputBufferBytes, finalReadEnd);
1032 EXPECT_EQ(td.inputBufferBytes % kUbBlockBytes, 0U);1205 EXPECT_EQ(td.inputBufferBytes % kUbBlockBytes, 0U);
1033 EXPECT_LE(td.inputBufferBytes * kDoubleBufferCount +1206 EXPECT_LE(td.inputBufferBytes * kDoubleBufferCount +