aclnnQuantize
Applicable Products
- Ascend 950PR&950DT products: Supported
- Atlas A3 products: Supported
- Atlas A2 products: Supported
- Atlas 200I/500 A2 inference products: Not supported
- Atlas inference products: Supported
- Atlas training products: Not supported
Function
-
Description: Quantizes the input tensor x.
-
Formula:
out=round((x/scales)+zeroPoints)out=round((x/scales)+zeroPoints)
Function Prototype
Each operator uses a two-phase API. Call the "aclnnQuantizeGetWorkspaceSize" API first to obtain the workspace size required for the computation and the executor containing the computation process, and then call the "aclnnQuantize" API to perform the computation.
aclnnStatus aclnnQuantizeGetWorkspaceSize(
const aclTensor* x,
const aclTensor* scales,
const aclTensor* zeroPoints,
aclDataType dtype,
int32_t axis,
aclTensor* out,
uint64_t* workspaceSize,
aclOpExecutor** executor)
aclnnStatus aclnnQuantize(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)
aclnnQuantizeGetWorkspaceSize
-
Parameters
Parameter Input/Output Description Instruction Data Type Data Format Dimension (shape) Non-contiguous Tensor x (aclTensor*) Input Source data tensor to perform quantization. Corresponds to `x` in the formula. Empty tensor is supported. FLOAT32, FLOAT16, BFLOAT16 ND 1-8 √ scales (aclTensor*) Input Scaling factor tensor in the quantization process. Corresponds to `scales` in the formula. - Supports empty tensors.
- The size must be 1 or equal to the size of the axis in input x.
- If the dtype of `x` is not FLOAT32, the dtype of scales must be consistent with the dtype of `x`.
FLOAT32, FLOAT16, BFLOAT16 ND 1 √ zeroPoints (aclTensor*) Input Represents the zero-point offset tensor in the quantization process, corresponding to `zeroPoints` in the formula. - Supports empty tensors.
- Supports passing in a null pointer.
- The size must be 1 or equal to the size of the axis in input x, and equal to the size of scales.
INT32, INT8, UINT8, FLOAT32, BFLOAT16 ND 1 √ dtype (aclDataType) Input Specifies the output data type. Can be set to ACL_INT8, ACL_UINT8, ACL_INT32, ACL_HIFLOAT8, ACL_FLOAT8_E4M3FN, or ACL_FLOAT8_E5M2. - - - - axis (int32_t) Input Represents the element-wise axis to perform quantization. Other axes are broadcast. - When the sizes of the input scales and zeroPoints are all 1, this parameter is not actually used.
- The supported range is negative values less than the dimension count of input x and greater than or equal to the negative of the dimension count of x.
- - - - out (aclTensor*) Output Represents the quantization output tensor. Corresponds to `out` in the corresponding formula. - Supports empty tensor.
- Its shape must be consistent with `x`, and the data type is specified by the input parameter `dtype`.
INT8, UINT8, INT32, HIFLOAT8, FLOAT8_E4M3FN, FLOAT8_E5M2 ND 1-8 √ workspaceSize (uint64_t*) Output Returns the size of the workspace to be requested on the Device side. - - - - - executor (aclOpExecutor**) Output Returns the op executor, which contains the operator computation process. - - - - - - Atlas inference products:
- Data type:
- The input parameters
xandscalesdo not support BFLOAT16 or FLOAT32. - The input parameter
zeroPointsdoes not support FLOAT32. When the data type is BFLOAT16, the data types ofxandscalesare both BFLOAT16. - The output
outsupports only INT8, UINT8, and INT32.
- The input parameters
- The input parameter
dtypesupports only ACL_INT8, ACL_UINT8, and ACL_INT32.
- Data type:
- Atlas A2 products, Atlas A3 products:
- Data type:
- The input parameter
zeroPointsdoes not support FLOAT32. When the data type is BFLOAT16, the data types ofxandscalesare both BFLOAT16. - The output
outsupports only INT8, UINT8, and INT32.
- The input parameter
- The input parameter
dtypesupports only ACL_INT8, ACL_UINT8, and ACL_INT32.
- Data type:
-
Return Value
aclnnStatus: return status code. For details, see aclnn Return Codes.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Error Code. Error Code Description ACLNN_ERR_PARAM_NULLPTR 161001 x, scales, or out is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type/data format/dimension of x, scales, zeroPoints, or out is out of the supported range. The axis specified by input axis exceeds the number of dimensions of input x. dtype is not within the valid value range. The sizes of input scales and zeroPoints are not equal. When the sizes of input scales and zeroPoints are not 1, they are not equal to the size of the axis specified by input axis. The data type of output out is inconsistent with the value of input dtype.
aclnnQuantize
-
Parameters
Parameter Input/Output Description workspace Input Workspace memory address allocated on the Device. workspaceSize Input Workspace size allocated on the Device, obtained by the first-phase API aclnnQuantizeGetWorkspaceSize. executor Input Op executor, which contains the operator computation flow. stream Input Stream for executing tasks. -
Return Value
aclnnStatus: return status code. For details, see aclnn Return Codes.
Constraints
- Deterministic computation:
- aclnnQuantize defaults to a deterministic implementation.
Example
The following sample code is for reference only. For details about compilation and execution, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_quantize.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape)
{
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
int Init(int32_t deviceId, aclrtStream* stream)
{
// Boilerplate for resource initialization
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(
const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr, aclDataType dataType,
aclTensor** tensor)
{
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to apply for memory on the device.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy data from the host to the device memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of the contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call aclCreateTensor to create an aclTensor.
*tensor = aclCreateTensor(
shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND, shape.data(), shape.size(),
*deviceAddr);
return 0;
}
int main()
{
// 1. (boilerplate) device/stream initialization. For details, see the acl API manual.
// Set deviceId based on the actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct the input and output. The construction is custom based on the API.
std::vector<int64_t> xShape = {4, 2};
std::vector<int64_t> scalesShape = {2};
std::vector<int64_t> zeroPointsShape = {2};
std::vector<int64_t> outShape = {4, 2};
void* xDeviceAddr = nullptr;
void* scalesDeviceAddr = nullptr;
void* zeroPointsDeviceAddr = nullptr;
void* outDeviceAddr = nullptr;
aclTensor* x = nullptr;
aclTensor* scales = nullptr;
aclTensor* zeroPoints = nullptr;
aclTensor* out = nullptr;
aclDataType dataType = ACL_INT32;
int32_t axis = 1;
std::vector<float> scalesHostData = {1.0, -3.0};
std::vector<int32_t> zeroPointsData = {2, 10};
std::vector<float> xHostData = {0.3382, -0.0919, 0.7564, 0.0234, 3.1024, 1.0761, 0.4228, 1.4621};
std::vector<int32_t> outHostData = {8, 0};
// Create the x aclTensor.
ret = CreateAclTensor(xHostData, xShape, &xDeviceAddr, aclDataType::ACL_FLOAT, &x);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the scales aclTensor.
ret = CreateAclTensor(scalesHostData, scalesShape, &scalesDeviceAddr, aclDataType::ACL_FLOAT, &scales);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the zeroPoints aclTensor.
ret = CreateAclTensor(zeroPointsData, zeroPointsShape, &zeroPointsDeviceAddr, aclDataType::ACL_INT32, &zeroPoints);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create out aclTensor
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, dataType, &out);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// 3. Call the CANN operator library API, and replace it with the specific API name
uint64_t workspaceSize = 0;
aclOpExecutor* executor;
// Call the first-phase API of aclnnQuantize
ret = aclnnQuantizeGetWorkspaceSize(x, scales, zeroPoints, dataType, axis, out, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantizeGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Allocate device memory based on the workspaceSize calculated by the first-phase API
void* workspaceAddr = nullptr;
if (workspaceSize > 0) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnQuantize
ret = aclnnQuantize(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnQuantize failed. ERROR: %d\n", ret); return ret);
// 4. (boilerplate) Synchronously wait for the task execution to complete
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value, copy the result on the device side to the host side, and modify it based on the definition of the specific API
auto outSize = GetShapeSize(outShape);
std::vector<int32_t> outData(outSize, 0);
ret = aclrtMemcpy(
outData.data(), outData.size() * sizeof(outData[0]), outDeviceAddr, outSize * sizeof(outData[0]),
ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return ret);
for (int64_t i = 0; i < outSize; i++) {
LOG_PRINT("out[%ld] is: %d\n", i, outData[i]);
}
// 6. Release aclTensor and aclScalar, and modify it based on the definition of the specific API
aclDestroyTensor(x);
aclDestroyTensor(scales);
aclDestroyTensor(zeroPoints);
aclDestroyTensor(out);
// 7. Release device resources, and modify it based on the definition of the specific API
aclrtFree(xDeviceAddr);
aclrtFree(scalesDeviceAddr);
aclrtFree(zeroPointsDeviceAddr);
aclrtFree(outDeviceAddr);
if (workspaceSize > 0) {
aclrtFree(workspaceAddr);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
return 0;
}