Merged
perf(in_training_reduce_v2): 优化短规约连续行计算 #13846
raoliang_sacAdd on 16 days ago
perf(in_training_reduce_v2): 优化短规约连续行计算 #13846
Merged
Total: 7 files changed+378-25
| @@ -39,20 +39,21 @@ English | [简体中文](./README.md) | |||
| 39 | | sum | Output | The result of summing `x` along the spatial dimensions. The instance dimension and the channel dimension are kept, and the size of each reduced dimension is 1. | FLOAT | NC1HWC0, NDC1HWC0, ND | | 39 | | sum | Output | The result of summing `x` along the spatial dimensions. The instance dimension and the channel dimension are kept, and the size of each reduced dimension is 1. | FLOAT | NC1HWC0, NDC1HWC0, ND | |
| 40 | | square_sum | Output | The result of the square sum of `x` along the spatial dimensions. The shape is the same as `sum`. | FLOAT | NC1HWC0, NDC1HWC0, ND | | 40 | | square_sum | Output | The result of the square sum of `x` along the spatial dimensions. The shape is the same as `sum`. | FLOAT | NC1HWC0, NDC1HWC0, ND | |
| 41 | 41 | ||
| 42 | -### Product Differences | ||
| 43 | - | ||
| 44 | -| Product | Static shape capability | Dynamic shape capability | Shape/rank and format mapping | | ||
| 45 | -| --- | --- | --- | --- | | ||
| 46 | -| <term>Ascend 950PR&950DT products</term> | `x` is NCHW, NCDHW, ND, NHWC, or NDHWC, and the two outputs are ND | Supports dynamic shapes of the preceding formats and ND dynamic rank. NHWC/NDHWC inputs keep the corresponding formats, NCHW/NCDHW/ND inputs use ND storage, and both outputs are ND | The NCHW input is `[N,C,H,W]` and the output is `[N,C,1,1]`; the NCDHW input is `[N,C,D,H,W]` and the output is `[N,C,1,1,1]`; the ND input has a rank of 2~8, the first two dimensions are N and C, and the remaining dimensions are reduced to 1; the NHWC input is `[N,H,W,C]` and the output is `[N,1,1,C]`; the NDHWC input is `[N,D,H,W,C]` and the output is `[N,1,1,1,C]` | | ||
| 47 | -| <term>Atlas A3 products</term><br><term>Atlas A2 products</term><br><term>Atlas 200I/500 A2 inference products</term><br><term>Atlas inference products</term><br><term>Atlas training products</term> | The 4D NCHW/NHWC original formats are mapped to NC1HWC0, and the 5D NCDHW/NDHWC original formats are mapped to NDC1HWC0. The storage formats of the input and the output are consistent | The registered dynamic shape and dynamic rank paths of the corresponding models use NC1HWC0 or NDC1HWC0 storage, and the format mapping is consistent with the static shape | The original input is 4D or 5D. The logical output keeps N and C, and H, W or D, H, W is 1. The storage format is mapped by the graph engine | | ||
| 48 | - | ||
| 49 | ## Constraints | 42 | ## Constraints |
| 50 | 43 | ||
| 51 | - The logical shape of NCHW/NHWC inputs is 4D, and the spatial dimensions are H and W. NCDHW/NDHWC inputs are 5D, and the spatial dimensions are D, H, and W. | 44 | - The logical shape of NCHW/NHWC inputs is 4D, and the spatial dimensions are H and W. NCDHW/NDHWC inputs are 5D, and the spatial dimensions are D, H, and W. |
| 52 | -- For the <term>Ascend 950PR&950DT products</term>, the ND input rank is 2~8. dim0 and dim1 are the instance dimension and the channel dimension respectively, and dim2 and later dimensions are spatial dimensions. | 45 | +- <term>Ascend 950PR&950DT products</term>: |
| 53 | -- The <term>Ascend 950PR&950DT products</term> require each dimension of `x` to be greater than 0. Empty tensors are not supported. | 46 | + - For static shapes, `x` is NCHW, NCDHW, ND, NHWC, or NDHWC, and the two outputs are ND. Dynamic shapes of these formats and ND dynamic rank are supported. NHWC/NDHWC inputs keep the corresponding formats, NCHW/NCDHW/ND inputs use ND storage, and both outputs are ND. |
| 54 | -- The <term>Ascend 950PR&950DT products</term> require the total number of input elements (the product of all dimension sizes) to not exceed `2^63-1`. | 47 | + - The NCHW input is `[N,C,H,W]` and the output is `[N,C,1,1]`; the NCDHW input is `[N,C,D,H,W]` and the output is `[N,C,1,1,1]`. |
| 55 | -- For the <term>Ascend 950PR&950DT products</term>, the channel row byte count `C*sizeof(x.dtype)` of NHWC/NDHWC inputs must not exceed `2^40-1`, that is, `C` of FLOAT16 must not exceed `2^39-1`, and `C` of FLOAT must not exceed `2^38-1`. | 48 | + - The NHWC input is `[N,H,W,C]` and the output is `[N,1,1,C]`; the NDHWC input is `[N,D,H,W,C]` and the output is `[N,1,1,1,C]`. |
| 49 | + - The ND input rank is 2~8. dim0 and dim1 are the instance dimension and the channel dimension respectively, and dim2 and later dimensions are spatial dimensions reduced to 1. | ||
| 50 | + - Each dimension of `x` must be greater than 0. Empty tensors are not supported. | ||
| 51 | + - The total number of input elements (the product of all dimension sizes) must not exceed `2^63-1`. | ||
| 52 | + - The channel row byte count `C*sizeof(x.dtype)` of NHWC/NDHWC inputs must not exceed `2^40-1`, that is, `C` of FLOAT16 must not exceed `2^39-1`, and `C` of FLOAT must not exceed `2^38-1`. | ||
| 53 | +- <term>Atlas A3 products</term>, <term>Atlas A2 products</term>, <term>Atlas 200I/500 A2 inference products</term>, <term>Atlas inference products</term>, and <term>Atlas training products</term>: | ||
| 54 | + - The 4D NCHW/NHWC original formats are mapped to NC1HWC0, and the 5D NCDHW/NDHWC original formats are mapped to NDC1HWC0. The storage formats of the input and the output are consistent. | ||
| 55 | + - The registered dynamic shape and dynamic rank paths of the corresponding models use NC1HWC0 or NDC1HWC0 storage, and the format mapping is consistent with the static shape. | ||
| 56 | + - The original input is 4D or 5D. The logical output keeps N and C, and H, W or D, H, W is 1. The storage format is mapped by the graph engine. | ||
| 56 | 57 | ||
| 57 | ## Invocation Description | 58 | ## Invocation Description |
| 58 | 59 | ||
| @@ -113,6 +113,7 @@ protected: | |||
| 113 | ge::graphStatus PostTiling() override; | 113 | ge::graphStatus PostTiling() override; |
| 114 | 114 | ||
| 115 | private: | 115 | private: |
| 116 | + bool DoPackedRowTiling(uint64_t elemSize); | ||
| 116 | // sub-R 分块 tiling(R 超单次 UB 容量) | 117 | // sub-R 分块 tiling(R 超单次 UB 容量) |
| 117 | bool DoSubRTiling(uint64_t rAlign, uint64_t binAddQuotient, int64_t elemSize); | 118 | bool DoSubRTiling(uint64_t rAlign, uint64_t binAddQuotient, int64_t elemSize); |
| 118 | // sub-R 路径 Kernel 侧 UB 精确占用(须与 op_kernel/arch35 的 InitSubR() 逐项一致) | 119 | // sub-R 路径 Kernel 侧 UB 精确占用(须与 op_kernel/arch35 的 InitSubR() 逐项一致) |
Mnorm/in_training_reduce_v2/op_host/arch35/in_training_reduce_v2_tiling_ar_full_reduce_arch35.cpp+46-0
| @@ -64,6 +64,13 @@ bool INTrainingReduceV2ARFullReduceTiling::IsCapable() | |||
| 64 | if (dataType == ge::DT_FLOAT16) { | 64 | if (dataType == ge::DT_FLOAT16) { |
| 65 | elemSize = FP16_BYTE; | 65 | elemSize = FP16_BYTE; |
| 66 | } | 66 | } |
| 67 | + if (r <= vlfp32 && DoPackedRowTiling(static_cast<uint64_t>(elemSize))) { | ||
| 68 | + return true; | ||
| 69 | + } | ||
| 70 | + if (r == 1) { | ||
| 71 | + OP_LOGE(GetNodeName(), "R=1 input/output buffers exceed UB capacity"); | ||
| 72 | + return false; | ||
| 73 | + } | ||
| 67 | 74 | ||
| 68 | const uint64_t rValue = static_cast<uint64_t>(r); | 75 | const uint64_t rValue = static_cast<uint64_t>(r); |
| 69 | const uint64_t elemSizeValue = static_cast<uint64_t>(elemSize); | 76 | const uint64_t elemSizeValue = static_cast<uint64_t>(elemSize); |
| @@ -171,6 +178,45 @@ bool INTrainingReduceV2ARFullReduceTiling::IsCapable() | |||
| 171 | return true; | 178 | return true; |
| 172 | } | 179 | } |
| 173 | 180 | ||
| 181 | +// Pack consecutive N*C rows without per-row DMA padding. For short reductions, | ||
| 182 | +// each vector lane owns one output row; R=1 keeps the elementwise fast path. | ||
| 183 | +bool INTrainingReduceV2ARFullReduceTiling::DoPackedRowTiling(uint64_t elemSize) | ||
| 184 | +{ | ||
| 185 | + const uint64_t usable = aicoreParams_.ubSize - RESERVE_FOR_ALIGN; | ||
| 186 | + const uint64_t vectorElements = static_cast<uint64_t>(vlfp32); | ||
| 187 | + const uint64_t rowBytes = static_cast<uint64_t>(r) * elemSize; | ||
| 188 | + const uint64_t bytesPerRow = DOUBLE_BUFFER_COUNT * (rowBytes + REDUCE_OUTPUT_COUNT * sizeof(float)); | ||
| 189 | + const uint64_t maxTileRows = usable / bytesPerRow / vectorElements * vectorElements; | ||
| 190 | + // Small mocked UB capacities can still use the original full-AR path. | ||
| 191 | + if (maxTileRows == 0) { | ||
| 192 | + return false; | ||
| 193 | + } | ||
| 194 | + const uint64_t balancedRows = Ops::Base::CeilAlign(Ops::Base::CeilDiv(totalRows_, aicoreParams_.blockDim), | ||
| 195 | + vectorElements); | ||
| 196 | + const uint64_t maxLoopRows = std::numeric_limits<uint16_t>::max() * vectorElements; | ||
| 197 | + const uint64_t maxCopyRows = std::numeric_limits<uint32_t>::max() / rowBytes / vectorElements * vectorElements; | ||
| 198 | + const uint64_t tileRows = std::min({maxTileRows, balancedRows, maxLoopRows, maxCopyRows}); | ||
| 199 | + const uint64_t totalTiles = Ops::Base::CeilDiv(totalRows_, tileRows); | ||
| 200 | + const uint64_t tilesPerCore = Ops::Base::CeilDiv(totalTiles, aicoreParams_.blockDim); | ||
| 201 | + blockNum_ = Ops::Base::CeilDiv(totalTiles, tilesPerCore); | ||
| 202 | + | ||
| 203 | + td_.numN = a1; | ||
| 204 | + td_.numC = a0; | ||
| 205 | + td_.numR = r; | ||
| 206 | + td_.rAlign = 0; // Packed rows use masked gathers, with no per-row padding. | ||
| 207 | + td_.cInner = static_cast<int64_t>(tileRows); | ||
| 208 | + td_.cOuter = static_cast<int64_t>(totalTiles); | ||
| 209 | + td_.cTail = static_cast<int64_t>(totalRows_ - (totalTiles - 1) * tileRows); | ||
| 210 | + td_.perCoreCnt = static_cast<int64_t>(tilesPerCore); | ||
| 211 | + td_.totalRows = static_cast<int64_t>(totalRows_); | ||
| 212 | + td_.totalElements = static_cast<int64_t>(totalElements_); | ||
| 213 | + td_.totalTiles = static_cast<int64_t>(totalTiles); | ||
| 214 | + // Full vector loads, including the last partial vector, stay inside UB. | ||
| 215 | + td_.inputBufferBytes = tileRows * rowBytes; | ||
| 216 | + td_.outputBufferBytes = tileRows * sizeof(float); | ||
| 217 | + return true; | ||
| 218 | +} | ||
| 219 | + | ||
| 174 | // sub-R 路径 Kernel 侧 UB 精确占用。逐项对应 InitSubR(): | 220 | // sub-R 路径 Kernel 侧 UB 精确占用。逐项对应 InitSubR(): |
| 175 | // inQueueX_ : CeilAlign(rFactor * elemSize, BLOCK_SIZE) * DOUBLE_BUFFER_NUM | 221 | // inQueueX_ : CeilAlign(rFactor * elemSize, BLOCK_SIZE) * DOUBLE_BUFFER_NUM |
| 176 | // sum/sqPartialBuf_ : CeilAlign(slots, VL_FP32) * sizeof(float) * ACC_COMPONENTS,两块 | 222 | // sum/sqPartialBuf_ : CeilAlign(slots, VL_FP32) * sizeof(float) * ACC_COMPONENTS,两块 |
| @@ -53,6 +53,83 @@ __simd_callee__ inline void ReduceFullAR(RegTensor<float>& result, RegTensor<flo | |||
| 53 | Add(result, values, low, one); | 53 | Add(result, values, low, one); |
| 54 | } | 54 | } |
| 55 | 55 | ||
| 56 | +template <typename T_X> | ||
| 57 | +__simd_vf__ inline void CalculateR1Vf(__ubuf__ T_X* xUb, __ubuf__ float* sumUb, __ubuf__ float* squareUb, | ||
| 58 | + uint32_t count, uint16_t loops) | ||
| 59 | +{ | ||
| 60 | + RegTensor<float> value; | ||
| 61 | + RegTensor<float> square; | ||
| 62 | + for (uint16_t i = 0; i < loops; ++i) { | ||
| 63 | + MaskReg valid = UpdateMask<float>(count); | ||
| 64 | + const uint32_t offset = static_cast<uint32_t>(i) * VL_FP32; | ||
| 65 | + if constexpr (IsSameType<T_X, float>::value) { | ||
| 66 | + AscendC::Reg::LoadAlign(value, xUb + offset); | ||
| 67 | + } else { | ||
| 68 | + RegTensor<T_X> input; | ||
| 69 | + AscendC::Reg::LoadAlign<T_X, LoadDist::DIST_UNPACK_B16>(input, xUb + offset); | ||
| 70 | + AscendC::Reg::Cast<float, T_X, castTraitB162B32>(value, input, valid); | ||
| 71 | + } | ||
| 72 | + Mul(square, value, value, valid); | ||
| 73 | + AscendC::Reg::StoreAlign(sumUb + offset, value, valid); | ||
| 74 | + AscendC::Reg::StoreAlign(squareUb + offset, square, valid); | ||
| 75 | + } | ||
| 76 | +} | ||
| 77 | + | ||
| 78 | +template <typename T_X> | ||
| 79 | +__simd_vf__ inline void CalculateShortRVf(__ubuf__ T_X* xUb, __ubuf__ float* sumUb, __ubuf__ float* squareUb, | ||
| 80 | + uint32_t rows, uint32_t reduceLen, uint16_t loops, float scale, float restore) | ||
| 81 | +{ | ||
| 82 | + RegTensor<uint32_t> rowOffsets; | ||
| 83 | + RegTensor<uint32_t> offsets; | ||
| 84 | + RegTensor<float> value; | ||
| 85 | + RegTensor<float> square; | ||
| 86 | + RegTensor<float> sumHi; | ||
| 87 | + RegTensor<float> sumLo; | ||
| 88 | + RegTensor<float> squareHi; | ||
| 89 | + RegTensor<float> squareLo; | ||
| 90 | + MaskReg full = CreateMask<float, MaskPattern::ALL>(); | ||
| 91 | + AscendC::Reg::Arange((RegTensor<int32_t>&)rowOffsets, static_cast<int32_t>(0)); | ||
| 92 | + Muls(rowOffsets, rowOffsets, reduceLen, full); | ||
| 93 | + for (uint16_t i = 0; i < loops; ++i) { | ||
| 94 | + const uint32_t rowOffset = static_cast<uint32_t>(i) * VL_FP32; | ||
| 95 | + uint32_t activeRows = rows; | ||
| 96 | + MaskReg valid = UpdateMask<float>(rows); | ||
| 97 | + activeRows -= rows; | ||
| 98 | + MaskReg validB16 = UpdateMask<half>(activeRows); | ||
| 99 | + Duplicate(sumHi, 0.0f, full); | ||
| 100 | + Duplicate(sumLo, 0.0f, full); | ||
| 101 | + Duplicate(squareHi, 0.0f, full); | ||
| 102 | + Duplicate(squareLo, 0.0f, full); | ||
| 103 | + __ubuf__ T_X* rowBase = xUb + rowOffset * reduceLen; | ||
| 104 | + for (uint16_t column = 0; column < reduceLen; ++column) { | ||
| 105 | + // ZEROING gives inactive lanes a safe zero index in the final tile. | ||
| 106 | + Adds(offsets, rowOffsets, static_cast<uint32_t>(column), valid); | ||
| 107 | + if constexpr (IsSameType<T_X, float>::value) { | ||
| 108 | + AscendC::Reg::Gather(value, rowBase, offsets, valid); | ||
| 109 | + } else { | ||
| 110 | + RegTensor<uint16_t> offsetsB16; | ||
| 111 | + RegTensor<T_X> gathered; | ||
| 112 | + RegTensor<T_X> unpacked; | ||
| 113 | + AscendC::Reg::Pack(offsetsB16, offsets); | ||
| 114 | + AscendC::Reg::Gather(gathered, rowBase, offsetsB16, validB16); | ||
| 115 | + AscendC::Reg::UnPack((RegTensor<uint32_t>&)unpacked, (RegTensor<uint16_t>&)gathered); | ||
| 116 | + AscendC::Reg::Cast<float, T_X, castTraitB162B32>(value, unpacked, valid); | ||
| 117 | + } | ||
| 118 | + Mul(square, value, value, valid); | ||
| 119 | + ScaleSumValue<T_X>(value, scale, valid); | ||
| 120 | + AccumulateWithError(sumHi, sumLo, value, valid); | ||
| 121 | + AccumulateWithError(squareHi, squareLo, square, valid); | ||
| 122 | + } | ||
| 123 | + ClearNonFiniteCorrection(sumLo, valid); | ||
| 124 | + ClearNonFiniteCorrection(squareLo, valid); | ||
| 125 | + Add(sumHi, sumHi, sumLo, valid); | ||
| 126 | + Add(squareHi, squareHi, squareLo, valid); | ||
| 127 | + ScaleSumValue<T_X>(sumHi, restore, valid); | ||
| 128 | + AscendC::Reg::StoreAlign(sumUb + rowOffset, sumHi, valid); | ||
| 129 | + AscendC::Reg::StoreAlign(squareUb + rowOffset, squareHi, valid); | ||
| 130 | + } | ||
| 131 | +} | ||
| 132 | + | ||
| 56 | template <typename T_X> | 133 | template <typename T_X> |
| 57 | class INTrainingReduceV2ARFullReduce { | 134 | class INTrainingReduceV2ARFullReduce { |
| 58 | public: | 135 | public: |
| @@ -102,6 +179,9 @@ public: | |||
| 102 | pipe_.InitBuffer(inQueueX_, DOUBLE_BUFFER_NUM, inputBufferBytes_); | 179 | pipe_.InitBuffer(inQueueX_, DOUBLE_BUFFER_NUM, inputBufferBytes_); |
| 103 | pipe_.InitBuffer(outQueueSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_); | 180 | pipe_.InitBuffer(outQueueSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_); |
| 104 | pipe_.InitBuffer(outQueueSquareSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_); | 181 | pipe_.InitBuffer(outQueueSquareSum_, DOUBLE_BUFFER_NUM, outputBufferBytes_); |
| 182 | + if (rAlign_ == 0) { | ||
| 183 | + return; | ||
| 184 | + } | ||
| 105 | pipe_.InitBuffer(binaryAddBuf_, scratchBufferBytes_); | 185 | pipe_.InitBuffer(binaryAddBuf_, scratchBufferBytes_); |
| 106 | pipe_.InitBuffer(squareBinaryAddBuf_, scratchBufferBytes_); | 186 | pipe_.InitBuffer(squareBinaryAddBuf_, scratchBufferBytes_); |
| 107 | } | 187 | } |
| @@ -123,6 +203,10 @@ public: | |||
| 123 | ProcessSubR(); | 203 | ProcessSubR(); |
| 124 | return; | 204 | return; |
| 125 | } | 205 | } |
| 206 | + if (rAlign_ == 0) { | ||
| 207 | + ProcessPackedRows(); | ||
| 208 | + return; | ||
| 209 | + } | ||
| 126 | const int64_t totalCnt = totalTiles_; | 210 | const int64_t totalCnt = totalTiles_; |
| 127 | const int64_t startIndex = blockIdx_ * perCoreCnt_; | 211 | const int64_t startIndex = blockIdx_ * perCoreCnt_; |
| 128 | if (startIndex >= totalCnt) { | 212 | if (startIndex >= totalCnt) { |
| @@ -152,6 +236,46 @@ public: | |||
| 152 | } | 236 | } |
| 153 | 237 | ||
| 154 | private: | 238 | private: |
| 239 | + __aicore__ inline void ProcessPackedRows() | ||
| 240 | + { | ||
| 241 | + const int64_t startIndex = blockIdx_ * perCoreCnt_; | ||
| 242 | + if (startIndex >= totalTiles_) { | ||
| 243 | + return; | ||
| 244 | + } | ||
| 245 | + const int64_t remaining = totalTiles_ - startIndex; | ||
| 246 | + const int64_t endIndex = startIndex + (perCoreCnt_ < remaining ? perCoreCnt_ : remaining); | ||
| 247 | + for (int64_t tile = startIndex; tile < endIndex; ++tile) { | ||
| 248 | + const int64_t offset = tile * cInner_; | ||
| 249 | + const uint32_t count = static_cast<uint32_t>(tile == totalTiles_ - 1 ? cTail_ : cInner_); | ||
| 250 | + // A single packed DMA replaces one padded DMA block per output row. | ||
| 251 | + const uint32_t inputCount = count * static_cast<uint32_t>(numR_); | ||
| 252 | + CopyInX(offset * numR_, 1, inputCount, inputCount); | ||
| 253 | + LocalTensor<T_X> xLocal = inQueueX_.DeQue<T_X>(); | ||
| 254 | + LocalTensor<TSum> sumLocal = outQueueSum_.AllocTensor<TSum>(); | ||
| 255 | + LocalTensor<TSum> squareSumLocal = outQueueSquareSum_.AllocTensor<TSum>(); | ||
| 256 | + CalculatePackedRows(xLocal, sumLocal, squareSumLocal, count); | ||
| 257 | + inQueueX_.FreeTensor(xLocal); | ||
| 258 | + outQueueSum_.EnQue(sumLocal); | ||
| 259 | + outQueueSquareSum_.EnQue(squareSumLocal); | ||
| 260 | + CopyOutSumSquareSum(offset, count); | ||
| 261 | + } | ||
| 262 | + } | ||
| 263 | + | ||
| 264 | + __aicore__ inline void CalculatePackedRows(const LocalTensor<T_X>& xLocal, LocalTensor<TSum>& sumLocal, | ||
| 265 | + LocalTensor<TSum>& squareSumLocal, uint32_t count) | ||
| 266 | + { | ||
| 267 | + __ubuf__ T_X* xUb = (__ubuf__ T_X*)xLocal.GetPhyAddr(); | ||
| 268 | + __ubuf__ float* sumUb = (__ubuf__ float*)sumLocal.GetPhyAddr(); | ||
| 269 | + __ubuf__ float* squareUb = (__ubuf__ float*)squareSumLocal.GetPhyAddr(); | ||
| 270 | + const uint16_t loops = static_cast<uint16_t>((count + VL_FP32 - 1) / VL_FP32); | ||
| 271 | + if (numR_ == 1) { | ||
| 272 | + asc_vf_call<CalculateR1Vf<T_X>>(xUb, sumUb, squareUb, count, loops); | ||
| 273 | + } else { | ||
| 274 | + asc_vf_call<CalculateShortRVf<T_X>>(xUb, sumUb, squareUb, count, static_cast<uint32_t>(numR_), loops, | ||
| 275 | + sumScale_, sumRestore_); | ||
| 276 | + } | ||
| 277 | + } | ||
| 278 | + | ||
| 155 | __aicore__ inline void CopyInX(int64_t offset, uint32_t cnt, uint32_t r, uint32_t rAlign) | 279 | __aicore__ inline void CopyInX(int64_t offset, uint32_t cnt, uint32_t r, uint32_t rAlign) |
| 156 | { | 280 | { |
| 157 | LocalTensor<T_X> xLocal = inQueueX_.AllocTensor<T_X>(); | 281 | LocalTensor<T_X> xLocal = inQueueX_.AllocTensor<T_X>(); |
| @@ -22,10 +22,10 @@ struct INTrainingReduceV2ARFullReduceTilingData { | |||
| 22 | int64_t numN; // N 轴(a1) | 22 | int64_t numN; // N 轴(a1) |
| 23 | int64_t numC; // C 轴(a0) | 23 | int64_t numC; // C 轴(a0) |
| 24 | int64_t numR; // 规约轴长度 R = H*W(5D: D*H*W) | 24 | int64_t numR; // 规约轴长度 R = H*W(5D: D*H*W) |
| 25 | - uint64_t rAlign; // R 按 dtype 元素宽度 32B 对齐后的长度 | 25 | + uint64_t rAlign; // R 的 32B 对齐长度;0 表示短 R 连续打包路径 |
| 26 | - int64_t cInner; // 单 tile 内一次处理的 C 行数(sub-R 分块路径恒为 1) | 26 | + int64_t cInner; // 单 tile 的 C 行数;打包路径为展平 N*C 后的行数;sub-R 恒为 1 |
| 27 | - int64_t cOuter; // C 轴切分的外层块数 | 27 | + int64_t cOuter; // C 轴块数;打包路径为展平 N*C 后的块数 |
| 28 | - int64_t cTail; // C 轴尾块长度 | 28 | + int64_t cTail; // C 轴尾块长度;打包路径为展平 N*C 后的尾块长度 |
| 29 | uint64_t binaryAddQuotient; // 小于 rAlign 的最大 2 的幂(pairwise 折叠点) | 29 | uint64_t binaryAddQuotient; // 小于 rAlign 的最大 2 的幂(pairwise 折叠点) |
| 30 | int64_t perCoreCnt; // 平均每个核处理的 tile 数 | 30 | int64_t perCoreCnt; // 平均每个核处理的 tile 数 |
| 31 | // ---- sub-R 分块路径(R 超单次 UB 容量时启用)---- | 31 | // ---- sub-R 分块路径(R 超单次 UB 容量时启用)---- |
| @@ -40,7 +40,7 @@ struct INTrainingReduceV2ARFullReduceTilingData { | |||
| 40 | uint64_t tailChunks; // 末组的 chunk 数 = numChunks - (numGroups-1)*chunksPerGroup | 40 | uint64_t tailChunks; // 末组的 chunk 数 = numChunks - (numGroups-1)*chunksPerGroup |
| 41 | int64_t totalRows; // N*C,由 host 做溢出校验后下发 | 41 | int64_t totalRows; // N*C,由 host 做溢出校验后下发 |
| 42 | int64_t totalElements; // N*C*R,由 host 做溢出校验后下发 | 42 | int64_t totalElements; // N*C*R,由 host 做溢出校验后下发 |
| 43 | - int64_t totalTiles; // full-load 路径的 N*cOuter;sub-R 路径等于 totalRows | 43 | + int64_t totalTiles; // full-load: N*cOuter;打包路径: cOuter;sub-R: totalRows |
| 44 | uint64_t inputBufferBytes; // inQueueX 单 buffer 字节数 | 44 | uint64_t inputBufferBytes; // inQueueX 单 buffer 字节数 |
| 45 | uint64_t outputBufferBytes; // sum/square_sum 各自单 buffer 字节数 | 45 | uint64_t outputBufferBytes; // sum/square_sum 各自单 buffer 字节数 |
| 46 | uint64_t scratchBufferBytes; // full-load 路径每个 binary-add scratch 的字节数 | 46 | uint64_t scratchBufferBytes; // full-load 路径每个 binary-add scratch 的字节数 |
| @@ -19,3 +19,11 @@ h31_smallc_float32_r63,UNKNOWN,in_training_reduce_v2,"((1, 1, 63, 7),)","('float | |||
| 19 | h31_smallc_float32_r64,UNKNOWN,in_training_reduce_v2,"((1, 1, 64, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 64, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),() | 19 | h31_smallc_float32_r64,UNKNOWN,in_training_reduce_v2,"((1, 1, 64, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 64, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),() |
| 20 | h31_smallc_float32_r65,UNKNOWN,in_training_reduce_v2,"((1, 1, 65, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 65, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),() | 20 | h31_smallc_float32_r65,UNKNOWN,in_training_reduce_v2,"((1, 1, 65, 7),)","('float32',)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('float32', 'float32')","('ND', 'ND')","((1, 1, 65, 7),)","('NHWC',)","((1, 1, 1, 7), (1, 1, 1, 7))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),() |
| 21 | h31_subr_float32_tail65,UNKNOWN,in_training_reduce_v2,"((1, 1, 131137),)","('float32',)","('ND',)","((1, 1, 1), (1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((1, 1, 131137),)","('ND',)","((1, 1, 1), (1, 1, 1))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),() | 21 | h31_subr_float32_tail65,UNKNOWN,in_training_reduce_v2,"((1, 1, 131137),)","('float32',)","('ND',)","((1, 1, 1), (1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((1, 1, 131137),)","('ND',)","((1, 1, 1), (1, 1, 1))","('ND', 'ND')",{},"((1, 2),)",,1e-08,(),(),True,Targeted check of reported tail-mask behavior; UpdateMask updates count by reference,,0,,(),() |
| 22 | +r1_flat_rows_fp16,UNKNOWN,in_training_reduce_v2,"((131, 1),)","('float16',)","('ND',)","((131, 1), (131, 1))","('float32', 'float32')","('ND', 'ND')","((131, 1),)","('ND',)","((131, 1), (131, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),() | ||
| 23 | +r1_flat_tail_fp32,UNKNOWN,in_training_reduce_v2,"((3, 65),)","('float32',)","('ND',)","((3, 65), (3, 65))","('float32', 'float32')","('ND', 'ND')","((3, 65),)","('ND',)","((3, 65), (3, 65))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),() | ||
| 24 | +r1_nchw_tail_fp16,UNKNOWN,in_training_reduce_v2,"((3, 65, 1, 1),)","('float16',)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 1, 1),)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),() | ||
| 25 | +r1_ncdhw_tail_fp32,UNKNOWN,in_training_reduce_v2,"((3, 65, 1, 1, 1),)","('float32',)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 1, 1, 1),)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,R=1 packed vector and multicore tail regression,,0,,(),() | ||
| 26 | +short_r7_rows131_fp16,UNKNOWN,in_training_reduce_v2,"((131, 1, 7),)","('float16',)","('ND',)","((131, 1, 1), (131, 1, 1))","('float32', 'float32')","('ND', 'ND')","((131, 1, 7),)","('ND',)","((131, 1, 1), (131, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),() | ||
| 27 | +short_r9_tail_fp32,UNKNOWN,in_training_reduce_v2,"((3, 65, 3, 3),)","('float32',)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 3, 3),)","('NCHW',)","((3, 65, 1, 1), (3, 65, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),() | ||
| 28 | +short_r63_ncdhw_fp16,UNKNOWN,in_training_reduce_v2,"((3, 65, 1, 7, 9),)","('float16',)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((3, 65, 1, 7, 9),)","('NCDHW',)","((3, 65, 1, 1, 1), (3, 65, 1, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),() | ||
| 29 | +short_r64_ndhwc_c1_fp32,UNKNOWN,in_training_reduce_v2,"((65, 4, 4, 4, 1),)","('float32',)","('NDHWC',)","((65, 1, 1, 1, 1), (65, 1, 1, 1, 1))","('float32', 'float32')","('ND', 'ND')","((65, 4, 4, 4, 1),)","('NDHWC',)","((65, 1, 1, 1, 1), (65, 1, 1, 1, 1))","('ND', 'ND')",{},"((-2, 2),)",,1e-08,(),(),True,Short reduction packed rows; vector/tile tails and compensated accumulation,,0,,(),() | ||
| @@ -70,7 +70,8 @@ static uint64_t RunTiling(gert::StorageShape& xShape, gert::StorageShape& outSha | |||
| 70 | INTrainingReduceV2ARFullReduceTilingData* tdOut = nullptr, | 70 | INTrainingReduceV2ARFullReduceTilingData* tdOut = nullptr, |
| 71 | ge::DataType sumDtype = ge::DT_FLOAT, ge::DataType squareSumDtype = ge::DT_FLOAT, | 71 | ge::DataType sumDtype = ge::DT_FLOAT, ge::DataType squareSumDtype = ge::DT_FLOAT, |
| 72 | uint64_t ubSize = 245760, INTrainingReduceV2ARAReduceTilingData* araOut = nullptr, | 72 | uint64_t ubSize = 245760, INTrainingReduceV2ARAReduceTilingData* araOut = nullptr, |
| 73 | - gert::StorageShape* squareSumShape = nullptr, ge::graphStatus* tilingStatus = nullptr) | 73 | + gert::StorageShape* squareSumShape = nullptr, ge::graphStatus* tilingStatus = nullptr, |
| 74 | + uint32_t* blockDimOut = nullptr) | ||
| 74 | { | 75 | { |
| 75 | std::map<std::string, std::string> socVersionInfos = {{"NpuArch", "3510"}}; | 76 | std::map<std::string, std::string> socVersionInfos = {{"NpuArch", "3510"}}; |
| 76 | map<string, string> socInfos; | 77 | map<string, string> socInfos; |
| @@ -153,6 +154,9 @@ static uint64_t RunTiling(gert::StorageShape& xShape, gert::StorageShape& outSha | |||
| 153 | if (status != ge::GRAPH_SUCCESS) { | 154 | if (status != ge::GRAPH_SUCCESS) { |
| 154 | return UINT64_MAX; | 155 | return UINT64_MAX; |
| 155 | } | 156 | } |
| 157 | + if (blockDimOut != nullptr) { | ||
| 158 | + *blockDimOut = tilingContext->GetBlockDim(); | ||
| 159 | + } | ||
| 156 | if (tdOut != nullptr) { | 160 | if (tdOut != nullptr) { |
| 157 | auto raw = tilingContext->GetRawTilingData(); | 161 | auto raw = tilingContext->GetRawTilingData(); |
| 158 | if (raw == nullptr || raw->GetDataSize() < sizeof(*tdOut)) { | 162 | if (raw == nullptr || raw->GetDataSize() < sizeof(*tdOut)) { |
| @@ -218,6 +222,173 @@ static void ExpectSubRSelfConsistent(const INTrainingReduceV2ARFullReduceTilingD | |||
| 218 | // --------------------------------------------------------------------------- | 222 | // --------------------------------------------------------------------------- |
| 219 | // AR_FULL_REDUCE:4D NCHW 典型 shape [4,16,32,32] → TilingKey 200000 | 223 | // AR_FULL_REDUCE:4D NCHW 典型 shape [4,16,32,32] → TilingKey 200000 |
| 220 | // --------------------------------------------------------------------------- | 224 | // --------------------------------------------------------------------------- |
| 225 | +TEST_F(INTrainingReduceV2TilingTest, R1PacksIndependentRowsAcrossInstanceBoundaries) | ||
| 226 | +{ | ||
| 227 | + const std::vector<std::vector<int64_t>> shapes = {{1, 1}, {1, 63}, {1, 64}, {1, 65}, | ||
| 228 | + {131, 1}, {120, 205}, {2144, 1962}}; | ||
| 229 | + for (auto dtype : {ge::DT_FLOAT16, ge::DT_FLOAT}) { | ||
| 230 | + const uint64_t elemSize = dtype == ge::DT_FLOAT16 ? sizeof(uint16_t) : sizeof(float); | ||
| 231 | + for (const auto& dims : shapes) { | ||
| 232 | + SCOPED_TRACE(::testing::Message() << "N=" << dims[0] << " C=" << dims[1] << " dtype=" << dtype); | ||
| 233 | + gert::StorageShape input = {{dims[0], dims[1]}, {dims[0], dims[1]}}; | ||
| 234 | + gert::StorageShape output = input; | ||
| 235 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 236 | + uint32_t blocks = 0; | ||
| 237 | + ASSERT_EQ(RunTiling(input, output, dtype, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, MOCK_UB_SIZE, | ||
| 238 | + nullptr, nullptr, nullptr, &blocks), | ||
| 239 | + EXPECTED_AR_TILING_KEY); | ||
| 240 | + ASSERT_EQ(td.numR, 1); | ||
| 241 | + ASSERT_EQ(td.totalRows, dims[0] * dims[1]); | ||
| 242 | + ASSERT_EQ(td.totalElements, td.totalRows); | ||
| 243 | + ASSERT_GT(td.cInner, 0); | ||
| 244 | + EXPECT_EQ(td.cInner % 64, 0); | ||
| 245 | + EXPECT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, td.totalRows); | ||
| 246 | + EXPECT_GT(td.cTail, 0); | ||
| 247 | + EXPECT_LE(td.cTail, td.cInner); | ||
| 248 | + EXPECT_GE(td.inputBufferBytes, static_cast<uint64_t>(td.cInner) * elemSize); | ||
| 249 | + EXPECT_GE(td.outputBufferBytes, static_cast<uint64_t>(td.cInner) * sizeof(float)); | ||
| 250 | + EXPECT_EQ(td.scratchBufferBytes, 0U); | ||
| 251 | + EXPECT_EQ(td.partialBufferBytes, 0U); | ||
| 252 | + EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, MOCK_UB_SIZE); | ||
| 253 | + ASSERT_GT(blocks, 0U); | ||
| 254 | + EXPECT_LE(blocks, 64U); | ||
| 255 | + EXPECT_LT((blocks - 1) * td.perCoreCnt, td.totalTiles); | ||
| 256 | + EXPECT_GE(blocks * td.perCoreCnt, td.totalTiles); | ||
| 257 | + if (dims[0] == 131 && dims[1] == 1) { | ||
| 258 | + // N must no longer force one padded reduction and DMA per row. | ||
| 259 | + EXPECT_GT(td.cInner, td.numC); | ||
| 260 | + EXPECT_EQ(td.totalTiles, 3); | ||
| 261 | + } | ||
| 262 | + } | ||
| 263 | + } | ||
| 264 | +} | ||
| 265 | + | ||
| 266 | +TEST_F(INTrainingReduceV2TilingTest, R1KeepsChannelFirstAndSingleChannelLayoutContracts) | ||
| 267 | +{ | ||
| 268 | + struct LayoutCase { | ||
| 269 | + std::vector<int64_t> dims; | ||
| 270 | + ge::Format format; | ||
| 271 | + }; | ||
| 272 | + const std::vector<LayoutCase> cases = { | ||
| 273 | + {{3, 65, 1}, ge::FORMAT_ND}, {{3, 65, 1, 1}, ge::FORMAT_NCHW}, | ||
| 274 | + {{3, 65, 1, 1, 1}, ge::FORMAT_NCDHW}, {{3, 65, 1, 1, 1, 1, 1, 1}, ge::FORMAT_ND}, | ||
| 275 | + {{131, 1, 1, 1}, ge::FORMAT_NHWC}, {{131, 1, 1, 1, 1}, ge::FORMAT_NDHWC}}; | ||
| 276 | + for (const auto& item : cases) { | ||
| 277 | + for (auto dtype : {ge::DT_FLOAT16, ge::DT_FLOAT}) { | ||
| 278 | + gert::Shape shape; | ||
| 279 | + for (int64_t dim : item.dims) { | ||
| 280 | + shape.AppendDim(dim); | ||
| 281 | + } | ||
| 282 | + gert::StorageShape input; | ||
| 283 | + input.MutableOriginShape() = shape; | ||
| 284 | + input.MutableStorageShape() = shape; | ||
| 285 | + gert::StorageShape output = input; | ||
| 286 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 287 | + ASSERT_EQ(RunTiling(input, output, dtype, item.format, &td), EXPECTED_AR_TILING_KEY); | ||
| 288 | + EXPECT_EQ(td.numR, 1); | ||
| 289 | + EXPECT_EQ(td.totalRows, item.dims[0] * item.dims[1]); | ||
| 290 | + EXPECT_EQ(td.totalTiles, td.cOuter); | ||
| 291 | + EXPECT_EQ(td.scratchBufferBytes, 0U); | ||
| 292 | + } | ||
| 293 | + } | ||
| 294 | +} | ||
| 295 | + | ||
| 296 | +TEST_F(INTrainingReduceV2TilingTest, R1Preserves64BitOffsetsAndBoundsVectorLoopCount) | ||
| 297 | +{ | ||
| 298 | + for (uint64_t ubSize : {MOCK_UB_SIZE, uint64_t{1} << 28}) { | ||
| 299 | + gert::StorageShape input = {{131073, 65537}, {131073, 65537}}; | ||
| 300 | + gert::StorageShape output = input; | ||
| 301 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 302 | + ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, ubSize), | ||
| 303 | + EXPECTED_AR_TILING_KEY); | ||
| 304 | + EXPECT_EQ(td.totalRows, 8590131201LL); | ||
| 305 | + EXPECT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, td.totalRows); | ||
| 306 | + EXPECT_GT((td.totalTiles - 1) * td.cInner, static_cast<int64_t>(UINT32_MAX)); | ||
| 307 | + EXPECT_LE(td.cInner / 64, std::numeric_limits<uint16_t>::max()); | ||
| 308 | + EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, ubSize); | ||
| 309 | + } | ||
| 310 | +} | ||
| 311 | + | ||
| 312 | +TEST_F(INTrainingReduceV2TilingTest, R1HonorsMinimumUbBudget) | ||
| 313 | +{ | ||
| 314 | + gert::StorageShape input = {{120, 205}, {120, 205}}; | ||
| 315 | + gert::StorageShape output = input; | ||
| 316 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 317 | + ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, 2048), | ||
| 318 | + EXPECTED_AR_TILING_KEY); | ||
| 319 | + EXPECT_EQ(td.cInner, 64); | ||
| 320 | + EXPECT_EQ(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, 2048U); | ||
| 321 | + EXPECT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, nullptr, ge::DT_FLOAT, ge::DT_FLOAT, 2047), | ||
| 322 | + UINT64_MAX); | ||
| 323 | +} | ||
| 324 | + | ||
| 325 | +TEST_F(INTrainingReduceV2TilingTest, ShortRPacksRowsWithinUbAndDmaBounds) | ||
| 326 | +{ | ||
| 327 | + for (auto dtype : {ge::DT_FLOAT16, ge::DT_FLOAT}) { | ||
| 328 | + const uint64_t elemSize = dtype == ge::DT_FLOAT16 ? sizeof(uint16_t) : sizeof(float); | ||
| 329 | + for (int64_t reduceSize : {2, 3, 7, 8, 9, 16, 32, 63, 64}) { | ||
| 330 | + for (int64_t rows : {1, 63, 64, 65, 131, 951603}) { | ||
| 331 | + SCOPED_TRACE(::testing::Message() << "rows=" << rows << " R=" << reduceSize << " dtype=" << dtype); | ||
| 332 | + gert::StorageShape input = {{rows, 1, reduceSize}, {rows, 1, reduceSize}}; | ||
| 333 | + gert::StorageShape output = {{rows, 1, 1}, {rows, 1, 1}}; | ||
| 334 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 335 | + uint32_t blocks = 0; | ||
| 336 | + ASSERT_EQ(RunTiling(input, output, dtype, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, MOCK_UB_SIZE, | ||
| 337 | + nullptr, nullptr, nullptr, &blocks), | ||
| 338 | + EXPECTED_AR_TILING_KEY); | ||
| 339 | + ASSERT_EQ(td.rAlign, 0U); | ||
| 340 | + EXPECT_EQ(td.isSubRTiling, 0U); | ||
| 341 | + ASSERT_GT(td.cInner, 0); | ||
| 342 | + EXPECT_EQ(td.cInner % 64, 0); | ||
| 343 | + EXPECT_EQ(td.totalRows, rows); | ||
| 344 | + EXPECT_EQ(td.totalElements, rows * reduceSize); | ||
| 345 | + EXPECT_EQ(td.totalTiles, td.cOuter); | ||
| 346 | + EXPECT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, rows); | ||
| 347 | + EXPECT_GT(td.cTail, 0); | ||
| 348 | + EXPECT_LE(td.cTail, td.cInner); | ||
| 349 | + EXPECT_EQ(td.inputBufferBytes, td.cInner * reduceSize * elemSize); | ||
| 350 | + EXPECT_EQ(td.outputBufferBytes, td.cInner * sizeof(float)); | ||
| 351 | + EXPECT_LE(td.inputBufferBytes, UINT32_MAX); | ||
| 352 | + EXPECT_EQ(td.scratchBufferBytes, 0U); | ||
| 353 | + EXPECT_EQ(td.partialBufferBytes, 0U); | ||
| 354 | + EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 512, MOCK_UB_SIZE); | ||
| 355 | + ASSERT_GT(blocks, 0U); | ||
| 356 | + EXPECT_LE(blocks, 64U); | ||
| 357 | + EXPECT_LT((blocks - 1) * td.perCoreCnt, td.totalTiles); | ||
| 358 | + EXPECT_GE(blocks * td.perCoreCnt, td.totalTiles); | ||
| 359 | + } | ||
| 360 | + } | ||
| 361 | + } | ||
| 362 | +} | ||
| 363 | + | ||
| 364 | +TEST_F(INTrainingReduceV2TilingTest, ShortRRetainsFullArFallbackAndLargeRowOffsets) | ||
| 365 | +{ | ||
| 366 | + for (int64_t reduceSize : {2, 64, 65}) { | ||
| 367 | + gert::StorageShape input = {{65537, 65537, reduceSize}, {65537, 65537, reduceSize}}; | ||
| 368 | + gert::StorageShape output = {{65537, 65537, 1}, {65537, 65537, 1}}; | ||
| 369 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 370 | + ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td), EXPECTED_AR_TILING_KEY); | ||
| 371 | + EXPECT_EQ(td.totalRows, 4295098369LL); | ||
| 372 | + EXPECT_EQ(td.totalElements, td.totalRows * reduceSize); | ||
| 373 | + if (reduceSize <= 64) { | ||
| 374 | + EXPECT_EQ(td.rAlign, 0U); | ||
| 375 | + EXPECT_GT((td.totalTiles - 1) * td.cInner * reduceSize, static_cast<int64_t>(UINT32_MAX)); | ||
| 376 | + } else { | ||
| 377 | + EXPECT_GT(td.rAlign, 0U); | ||
| 378 | + EXPECT_EQ(td.totalTiles, td.numN * td.cOuter); | ||
| 379 | + } | ||
| 380 | + } | ||
| 381 | + gert::StorageShape input = {{131, 1, 2}, {131, 1, 2}}; | ||
| 382 | + gert::StorageShape output = {{131, 1, 1}, {131, 1, 1}}; | ||
| 383 | + INTrainingReduceV2ARFullReduceTilingData td{}; | ||
| 384 | + ASSERT_EQ(RunTiling(input, output, ge::DT_FLOAT, ge::FORMAT_ND, &td, ge::DT_FLOAT, ge::DT_FLOAT, 2048), | ||
| 385 | + EXPECTED_AR_TILING_KEY); | ||
| 386 | + // A packed 64-row tile needs 2560 bytes; the original full-AR route still fits. | ||
| 387 | + EXPECT_GT(td.rAlign, 0U); | ||
| 388 | + EXPECT_EQ(td.isSubRTiling, 0U); | ||
| 389 | + EXPECT_LE(2 * td.inputBufferBytes + 4 * td.outputBufferBytes + 2 * td.scratchBufferBytes, 2048U); | ||
| 390 | +} | ||
| 391 | + | ||
| 221 | TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nchw_200000_001) | 392 | TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nchw_200000_001) |
| 222 | { | 393 | { |
| 223 | gert::StorageShape x_shape = {{4, 16, 32, 32}, {4, 16, 32, 32}}; | 394 | gert::StorageShape x_shape = {{4, 16, 32, 32}, {4, 16, 32, 32}}; |
| @@ -588,9 +759,7 @@ TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_sub_r_nc_over_uint32_ | |||
| 588 | // --------------------------------------------------------------------------- | 759 | // --------------------------------------------------------------------------- |
| 589 | // N*C 溢出 uint32 且 R 很小时,full-load 路径同样必须被接受。 | 760 | // N*C 溢出 uint32 且 R 很小时,full-load 路径同样必须被接受。 |
| 590 | // 本例 N=65536, C=65536(N*C=2^32 > UINT32_MAX)、R=4: | 761 | // 本例 N=65536, C=65536(N*C=2^32 > UINT32_MAX)、R=4: |
| 591 | -// rAlign = CeilAlign(4*4, 32)/4 = 8 | 762 | +// 短规约按连续输出行打包;tiling 和 GM 行偏移全程 64 位,无截断。 |
| 592 | -// cInner = (245760-512) / (8*4*2 + 4*2*2 + 32*2) = 245248/144 = 1703 >= 1 | ||
| 593 | -// 故走 full-load;tiling 全程 64 位,无截断。 | ||
| 594 | // --------------------------------------------------------------------------- | 763 | // --------------------------------------------------------------------------- |
| 595 | TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nc_over_uint32_small_r_accepted_019) | 764 | TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nc_over_uint32_small_r_accepted_019) |
| 596 | { | 765 | { |
| @@ -609,7 +778,9 @@ TEST_F(INTrainingReduceV2TilingTest, tiling_ar_full_reduce_nc_over_uint32_small_ | |||
| 609 | ASSERT_EQ(td.numN * td.numC, 4294967296LL); | 778 | ASSERT_EQ(td.numN * td.numC, 4294967296LL); |
| 610 | ASSERT_EQ(td.totalRows, 4294967296LL); | 779 | ASSERT_EQ(td.totalRows, 4294967296LL); |
| 611 | ASSERT_EQ(td.totalElements, 17179869184LL); | 780 | ASSERT_EQ(td.totalElements, 17179869184LL); |
| 612 | - ASSERT_EQ(td.totalTiles, td.numN * td.cOuter); | 781 | + ASSERT_EQ(td.rAlign, 0U); |
| 782 | + ASSERT_EQ(td.totalTiles, td.cOuter); | ||
| 783 | + ASSERT_EQ((td.totalTiles - 1) * td.cInner + td.cTail, td.totalRows); | ||
| 613 | } | 784 | } |
| 614 | 785 | ||
| 615 | // --------------------------------------------------------------------------- | 786 | // --------------------------------------------------------------------------- |
| @@ -1024,10 +1195,12 @@ TEST_F(INTrainingReduceV2TilingTest, ArLastVectorLoadFitsInputAllocation) | |||
| 1024 | INTrainingReduceV2ARFullReduceTilingData td{}; | 1195 | INTrainingReduceV2ARFullReduceTilingData td{}; |
| 1025 | ASSERT_EQ(RunTiling(xShape, outShape, dtype, ge::FORMAT_NCHW, &td), 200000U); | 1196 | ASSERT_EQ(RunTiling(xShape, outShape, dtype, ge::FORMAT_NCHW, &td), 200000U); |
| 1026 | ASSERT_EQ(td.isSubRTiling, 0U); | 1197 | ASSERT_EQ(td.isSubRTiling, 0U); |
| 1027 | - const uint64_t finalReadEnd = ((td.cInner - 1U) * td.rAlign + | 1198 | + const uint64_t finalReadEnd = td.rAlign == 0 ? |
| 1028 | - (static_cast<uint64_t>(reduceSize) + kVectorElementsFp32 - 1U) / | 1199 | + td.cInner * reduceSize * elemSize : |
| 1029 | - kVectorElementsFp32 * kVectorElementsFp32) * | 1200 | + ((td.cInner - 1U) * td.rAlign + |
| 1030 | - elemSize; | 1201 | + (static_cast<uint64_t>(reduceSize) + kVectorElementsFp32 - 1U) / |
| 1202 | + kVectorElementsFp32 * kVectorElementsFp32) * | ||
| 1203 | + elemSize; | ||
| 1031 | EXPECT_GE(td.inputBufferBytes, finalReadEnd); | 1204 | EXPECT_GE(td.inputBufferBytes, finalReadEnd); |
| 1032 | EXPECT_EQ(td.inputBufferBytes % kUbBlockBytes, 0U); | 1205 | EXPECT_EQ(td.inputBufferBytes % kUbBlockBytes, 0U); |
| 1033 | EXPECT_LE(td.inputBufferBytes * kDoubleBufferCount + | 1206 | EXPECT_LE(td.inputBufferBytes * kDoubleBufferCount + |