aclnnAddLora
Applicable Products
- Ascend 950PR&950DT products: not supported
- Atlas A3 products: not supported
- Atlas A2 products: supported
- Atlas 200I/500 A2 inference products: not supported
- Atlas inference products: supported
- Atlas training products: not supported
Function
-
Description:
Multiplies the input x with the corresponding weightA and weightB based on the input indices, and then accumulates the result onto the input y and outputs it.
-
Formula:
Given an input tensor x whose last dimension has a length of 2d, the AddLora function performs the following computation:
-
Rearrange x according to the indices in indices, so that x corresponding to the same group of weights are placed together.
-
Iterate over each LoRA group, and perform matrix multiplication on the corresponding x and weightA respectively:
Z1=xi⋅weightA[i,layerIdx,:,:]Z1 = x_{i} \cdot weightA[i, layerIdx, :, :]
-
The resulting
Z1then undergoes matrix multiplication with weightB:Z2=Z1⋅weightB[i,layerIdx,:,:]×scaleZ2 = Z1 \cdot weightB[i, layerIdx, :, :] \times scale
-
Finally, accumulate the output
Z2onto y:out=y[:,yOffset:yOffset+ySliceSize]+Z2\text{out} = y[:, yOffset: yOffset+ySliceSize] + Z2
-
Function Prototype
Each operator uses a two-phase API. Call the "aclnnAddLoraGetWorkspaceSize" API first to obtain the input parameters and calculate the required workspace size based on the calculation process, and then call the "aclnnAddLora" API to perform the computation.
aclnnStatus aclnnAddLoraGetWorkspaceSize(
const aclTensor *y,
const aclTensor *x,
const aclTensor *weightB,
const aclTensor *indices,
const aclTensor *weightAOptional,
int64_t layerIdx,
double scale,
int64_t yOffset,
int64_t ySliceSize,
const aclTensor *out,
uint64_t *workspaceSize,
aclOpExecutor **executor)
aclnnStatus aclnnAddLora(
void *workspace,
uint64_t workspaceSize,
aclOpExecutor *executor,
aclrtStream stream)
aclnnAddLoraGetWorkspaceSize
-
Parameters:
Parameter Input/Output Description Instruction Data Type Data Format Dimension (shape) Non-contiguous Tensor y (aclTensor*) Input Indicates the tensor to be updated by accumulation, y in the formula. - 2D shape: [B, H3]. H3 must be an integer multiple of 16, and the range of H3 must support 1-131072.
- The first dimension needs to be consistent with the first dimension of x, both represented by `B`.
- Does not support empty tensors.
FLOAT16 ND 2 √ x (aclTensor*) Input Indicates the input tensor before grouping, x in the formula. - 2 dimensions in shape: [B, H1], and H1 is an integer multiple of 16.
- Does not support empty tensors.
FLOAT16 ND 2 √ weightB (aclTensor*) Input Indicates the second weight matrix for matrix multiplication, that is, weightB in the formula. - 4 dimensions in shape: [W, L, H2, R]. The value of the third dimension needs to be less than or equal to the second dimension of y (H2 ≤ H3), and H2 is an integer multiple of 16; the range of H2 must support 1-131072. The range of R must support 1-128, and R is an integer multiple of 16.
- Does not support empty tensors.
FLOAT16 ND, NZ 4 √ indices (aclTensor*) Input Identifies the group index of input x, that is, the input indices in the formula. - 1-D shape: [B].
- The first dimension must keep consistent with the first dimensions of x and y, both denoted as `B`.
- This parameter does not support empty tensors.
INT32 ND 1 √ weightAOptional (aclTensor*) Input Indicates the first weight matrix for matrix multiplication. If it is empty, the first matrix multiplication is skipped. weightA in the formula. - shape with 4 dimensions: [W, L, R, H1]. The first two dimensions must keep consistent with the first two dimensions of `weightB`, represented by `W` and `L`, where the range of W is 1-32 and the range of L is 1-32. The third dimension must keep consistent with the fourth dimension of `weightB`, both represented by `R`. The fourth dimension must keep consistent with the second dimension of `x`, both represented by `H1`, and must be an integer multiple of 16.
- Does not support empty tensors.
FLOAT16 ND, NZ √ layerIdx (int64_t) Input Indicates the layer index, layerIdx in the formula. The value needs to be less than the second dimension L of weightB. - - - - scale (double) Input Indicates the scaling coefficient, that is, scale in the formula. - - - - - yOffset (int64_t) Input Indicates the offset for updating y, that is, yOffset in the formula. The value needs to be less than the second dimension H3 of y. - - - - ySliceSize (int64_t) Input Indicates the range for updating y, namely ySliceSize in the formula. The value needs to be less than or equal to the second dimension H3 of y. - - - - out(aclTensor*) Output Output tensor, the output out in the formula. - The data type of the output must keep consistent with that of the input.
- The dimensions of the output shape must keep consistent with those of the input y shape.
FLOAT16 ND 2 √ workspaceSize(uint64_t*) Output Returns the workspace size that needs to be applied on the Device side. - - - - - executor (aclOpExecutor**) Output Returns the op executor containing the operator computation flow. - - - - - - Atlas A2 products: weightB and weightAOptional support the ND data format.
-
Return value
aclnnStatus: return status code. For details, see aclnn Return Codes.
The first-phase API performs input parameter validation and returns an error in the following scenarios:
Error Code Error code Description ACLNN_ERR_PARAM_NULLPTR 161001 The passed input parameter (x, y, weightB, indices) or output parameter out is a null pointer. ACLNN_ERR_PARAM_INVALID 161002 The data type of the input parameter (x, y, weightB, indices) or output parameter is not within the supported range. ACLNN_ERR_INNER_TILING_ERROR 561002 The shape information of the input tensors does not match. (For details, see Parameters.) The shape information of the input tensor is currently not supported. (For details, see Parameters.)
aclnnAddLora
-
Parameters:
Parameter Input/Output Description workspace Input Workspace memory address applied on the Device side. workspaceSize Input Workspace size applied on the Device side, obtained by the first-phase API aclnnAddLoraGetWorkspaceSize. executor Input Op executor that contains the operator computation flow. stream Input Stream for executing tasks. -
Return value
aclnnStatus: return status code. For details, see aclnn Return Codes.
Constraints
- Deterministic computation:
- aclnnAddLora defaults to a deterministic implementation.
Example
The sample code is as follows and is for reference only. For details about compilation and execution, see Compile and Run Samples.
#include <iostream>
#include <vector>
#include "acl/acl.h"
#include "aclnnop/aclnn_add_lora.h"
#define CHECK_RET(cond, return_expr) \
do { \
if (!(cond)) { \
return_expr; \
} \
} while (0)
#define LOG_PRINT(message, ...) \
do { \
printf(message, ##__VA_ARGS__); \
} while (0)
int64_t GetShapeSize(const std::vector<int64_t>& shape) {
int64_t shapeSize = 1;
for (auto i : shape) {
shapeSize *= i;
}
return shapeSize;
}
void PrintOutResult(std::vector<int64_t> &shape, void** deviceAddr) {
auto size = GetShapeSize(shape);
std::vector<float> resultData(size, 0);
auto ret = aclrtMemcpy(resultData.data(), resultData.size() * sizeof(resultData[0]),
*deviceAddr, size * sizeof(resultData[0]), ACL_MEMCPY_DEVICE_TO_HOST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("copy result from device to host failed. ERROR: %d\n", ret); return);
for (int64_t i = 0; i < size; i++) {
LOG_PRINT("result[%ld] is: %f\n", i, resultData[i]);
}
}
int Init(int32_t deviceId, aclrtStream* stream) {
// Boilerplate, initialization.
auto ret = aclInit(nullptr);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclInit failed. ERROR: %d\n", ret); return ret);
ret = aclrtSetDevice(deviceId);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSetDevice failed. ERROR: %d\n", ret); return ret);
ret = aclrtCreateStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtCreateStream failed. ERROR: %d\n", ret); return ret);
return 0;
}
template <typename T>
int CreateAclTensor(const std::vector<T>& hostData, const std::vector<int64_t>& shape, void** deviceAddr,
aclDataType dataType, aclTensor** tensor) {
auto size = GetShapeSize(shape) * sizeof(T);
// Call aclrtMalloc to allocate device-side memory.
auto ret = aclrtMalloc(deviceAddr, size, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMalloc failed. ERROR: %d\n", ret); return ret);
// Call aclrtMemcpy to copy host-side data to device-side memory.
ret = aclrtMemcpy(*deviceAddr, size, hostData.data(), size, ACL_MEMCPY_HOST_TO_DEVICE);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtMemcpy failed. ERROR: %d\n", ret); return ret);
// Calculate the strides of the contiguous tensor.
std::vector<int64_t> strides(shape.size(), 1);
for (int64_t i = shape.size() - 2; i >= 0; i--) {
strides[i] = shape[i + 1] * strides[i + 1];
}
// Call the aclCreateTensor API to create an aclTensor.
*tensor = aclCreateTensor(shape.data(), shape.size(), dataType, strides.data(), 0, aclFormat::ACL_FORMAT_ND,
shape.data(), shape.size(), *deviceAddr);
return 0;
}
int main() {
// 1. (Boilerplate) Initialize device/stream. For details, see the acl API documentation.
// Set deviceId based on the actual device.
int32_t deviceId = 0;
aclrtStream stream;
auto ret = Init(deviceId, &stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("Init acl failed. ERROR: %d\n", ret); return ret);
// 2. Construct inputs and outputs as required by the API.
int32_t batchSize = 1;
int32_t H1 = 16;
int32_t H2 = 16;
int32_t R = 16;
int32_t loraNum = 1;
int32_t layerNum = 1;
std::vector<int64_t> xShape = {batchSize, H1};
std::vector<int64_t> yShape = {batchSize, H2};
std::vector<int64_t> weightBShape = {loraNum, layerNum, H2, R};
std::vector<int64_t> indicesShape = {batchSize};
std::vector<int64_t> weightAShape = {loraNum, layerNum, R, H1};
std::vector<int64_t> outShape = {batchSize, H2};
std::vector<float> xHostData(batchSize * H1, 1);
std::vector<float> yHostData(batchSize * H2, 1);
std::vector<float> weightBHostData(loraNum * layerNum * H2 * R, 1);
std::vector<float> indicesHostData(batchSize, 0);
std::vector<float> weightAHostData(loraNum * layerNum * R * H1, 1);
std::vector<float> outHostData(batchSize * H2, 1);
void* xInputDeviceAddr = nullptr;
void* yInputDeviceAddr = nullptr;
void* weightBInputDeviceAddr = nullptr;
void* indicesInputDeviceAddr = nullptr;
void* weightAInputDeviceAddr = nullptr;
void* outDeviceAddr = nullptr;
aclTensor* xInput = nullptr;
aclTensor* yInput = nullptr;
aclTensor* weightBInput = nullptr;
aclTensor* indicesInput = nullptr;
aclTensor* weightAInput = nullptr;
aclTensor* out = nullptr;
// Create input x.
ret = CreateAclTensor(xHostData, xShape, &xInputDeviceAddr, aclDataType::ACL_FLOAT16, &xInput);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create input y.
ret = CreateAclTensor(yHostData, yShape, &yInputDeviceAddr, aclDataType::ACL_FLOAT16, &yInput);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create input weightB.
ret = CreateAclTensor(weightBHostData, weightBShape, &weightBInputDeviceAddr, aclDataType::ACL_FLOAT16, &weightBInput);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create input indices.
ret = CreateAclTensor(indicesHostData, indicesShape, &indicesInputDeviceAddr, aclDataType::ACL_INT32, &indicesInput);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create input weightA.
ret = CreateAclTensor(weightAHostData, weightAShape, &weightAInputDeviceAddr, aclDataType::ACL_FLOAT16, &weightAInput);
CHECK_RET(ret == ACL_SUCCESS, return ret);
// Create the out aclTensor.
ret = CreateAclTensor(outHostData, outShape, &outDeviceAddr, aclDataType::ACL_FLOAT16, &out);
CHECK_RET(ret == ACL_SUCCESS, return ret);
int64_t layer_idx = 0;
double scale = 1.0;
int64_t y_offset = 0;
int64_t y_slice_size = H2;
// 3. Call the CANN operator library API. Replace it with the specific API name.
uint64_t workspaceSize = 16 * 1024 * 1024;
aclOpExecutor* executor;
// Call the first-phase API of aclnnAddLora.
ret = aclnnAddLoraGetWorkspaceSize(yInput, xInput, weightBInput, indicesInput, weightAInput, layer_idx, scale, y_offset, y_slice_size, out, &workspaceSize, &executor);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddLoraGetWorkspaceSize failed. ERROR: %d\n", ret); return ret);
// Allocate device memory based on the workspaceSize calculated by the first-phase API.
void* workspaceAddr = nullptr;
if (workspaceSize > static_cast<uint64_t>(0)) {
ret = aclrtMalloc(&workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("allocate workspace failed. ERROR: %d\n", ret); return ret);
}
// Call the second-phase API of aclnnAddLora.
ret = aclnnAddLora(workspaceAddr, workspaceSize, executor, stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclnnAddLora failed. ERROR: %d\n", ret); return ret);
// 4. (boilerplate) Synchronously wait for the task execution to complete.
ret = aclrtSynchronizeStream(stream);
CHECK_RET(ret == ACL_SUCCESS, LOG_PRINT("aclrtSynchronizeStream failed. ERROR: %d\n", ret); return ret);
// 5. Obtain the output value and copy the result from the device side to the host side. Modify it based on the specific API definition.
PrintOutResult(outShape, &outDeviceAddr);
// 6. Release aclTensor. Modify it based on the specific API definition.
aclDestroyTensor(xInput);
aclDestroyTensor(yInput);
aclDestroyTensor(weightBInput);
aclDestroyTensor(indicesInput);
aclDestroyTensor(weightAInput);
aclDestroyTensor(out);
// 7. Release device resources. Modify the code according to the specific API definition.
aclrtFree(xInputDeviceAddr);
aclrtFree(yInputDeviceAddr);
aclrtFree(weightBInputDeviceAddr);
aclrtFree(indicesInputDeviceAddr);
aclrtFree(weightAInputDeviceAddr);
aclrtFree(outDeviceAddr);
if (workspaceSize > static_cast<uint64_t>(0)) {
aclrtFree(workspaceAddr);
}
aclrtDestroyStream(stream);
aclrtResetDevice(deviceId);
aclFinalize();
return 0;
}