Using MindSpore Lite for Image Classification (ArkTS)
When to Use
You can use @ohos.ai.mindSporeLite to quickly deploy AI algorithms into your application to perform AI model inference for image classification.
Image classification can be used to recognize objects in images and is widely used in areas such as medical image analysis, auto driving, e-commerce, and facial recognition.
Basic Concepts
Before getting started, you need to understand the following basic concepts:
Tensor: a special data structure that is similar to an array or matrix. It is the basic data structure used in MindSpore Lite network operations.
Float16 inference mode: an inference mode in half-precision format, where a number is represented with 16 bits.
Available APIs
APIs involved in MindSpore Lite model inference are categorized into context APIs, model APIs, and tensor APIs. For details about APIs, see @ohos.ai.mindSporeLite.
| API | Description |
|---|---|
| loadModelFromFile(model: string, context?: Context): Promise<Model> | Loads a model from a file. |
| getInputs(): MSTensor[] | Obtains the model input. |
| predict(inputs: MSTensor[]): Promise<MSTensor[]> | Performs model inference. |
| getData(): ArrayBuffer | Obtains tensor data. |
| setData(inputArray: ArrayBuffer): void | Sets tensor data. |
Development Process
- Select an image classification model.
- Use the MindSpore Lite inference model on the device to classify the selected images.
How to Develop
The following uses inference on an image in the album as an example to describe how to use MindSpore Lite to implement image classification.
Selecting a Model
This sample application uses mobilenetv2.ms as the image classification model. The model file is available in the entry/src/main/resources/rawfile project directory.
If you have other pre-trained models for image classification, convert the original model into the .ms format by referring to Using MindSpore Lite for Model Conversion.
Writing Inference Code
-
If the capability set defined by the project does not contain MindSpore Lite, create the syscap.json file in the entry/src/main directory of the DevEco Studio project. The file content is as follows:
{ "devices": { "general": [ // The value must be the same as the value of deviceTypes in the module.json5 file. "default" ] }, "development": { "addedSysCaps": [ "SystemCapability.AI.MindSporeLite" ] } } -
Call @ohos.ai.mindSporeLite to implement inference on the device. The operation process is as follows:
- Create a context, and set parameters such as the number of runtime threads and device type. The sample model does not support NNRt inference.
- Load the model. In this example, the model is loaded from the memory.
- Load data. Before executing a model, you need to obtain the model input and then fill data in the input tensors.
- Perform model inference through the predict API.
// model.ets import { mindSporeLite } from '@kit.MindSporeLiteKit' import { hilog } from '@kit.PerformanceAnalysisKit'; export default async function modelPredict( modelBuffer: ArrayBuffer, inputsBuffer: ArrayBuffer[]): Promise<mindSporeLite.MSTensor[]> { // 1. Create a context, and set parameters such as the number of runtime threads and device type. The context.target = ["nnrt"] option is not supported. let context: mindSporeLite.Context = {}; context.target = ['cpu']; context.cpu = {} context.cpu.threadNum = 2; context.cpu.threadAffinityMode = 1; context.cpu.precisionMode = 'enforce_fp32'; // 2. Load the model from the memory. let msLiteModel: mindSporeLite.Model = await mindSporeLite.loadModelFromBuffer(modelBuffer, context); // 3. Set the input data. let modelInputs: mindSporeLite.MSTensor[] = msLiteModel.getInputs(); for (let i = 0; i < inputsBuffer.length; i++) { let inputBuffer = inputsBuffer[i]; if (inputBuffer != null) { modelInputs[i].setData(inputBuffer as ArrayBuffer); } } // 4. Perform inference. hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `=========MS_LITE_LOG: MS_LITE predict start=====`); let modelOutputs: mindSporeLite.MSTensor[] = await msLiteModel.predict(modelInputs); return modelOutputs; }
Implementing Image Input and Preprocessing and Performing Inference
-
Call @ohos.file.picker to pick up the desired image in the album.
-
Based on the input image size, call @ohos.multimedia.image and @ohos.file.fs to perform operations such as cropping the image, obtaining the image buffer, and standardizing the image.
-
Load the model file and call the inference function to perform inference on the selected image, and process the inference result.
// Index.ets import modelPredict from './model'; import { photoAccessHelper } from '@kit.MediaLibraryKit'; import { BusinessError } from '@kit.BasicServicesKit'; import { image } from '@kit.ImageKit'; import { fileIo } from '@kit.CoreFileKit'; import { abilityAccessCtrl, Permissions } from '@kit.AbilityKit'; import { hilog } from '@kit.PerformanceAnalysisKit'; const PERMISSIONS: Permissions[] = ['ohos.permission.READ_IMAGEVIDEO']; @Entry @Component struct Index { @State modelPredict: string = 'MindSporeLite ArkTS Demo'; @State modelName: string = 'mobilenetv2.ms'; @State modelInputHeight: number = 224; @State modelInputWidth: number = 224; @State uris: Array<string> = []; @State max: number = 0; @State maxIndex: number = 0; @State maxArray: Array<number> = []; @State maxIndexArray: Array<number> = []; build() { Row() { Column() { Text(this.modelPredict) Button() { Text('photo') .fontSize(30) .fontWeight(FontWeight.Bold) } .onClick(() => { let resMgr = this.getUIContext()?.getHostContext()?.getApplicationContext().resourceManager; if (resMgr === null || resMgr === undefined){ hilog.error(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_ERR: get resMgr failed.`); return } resMgr?.getRawFileContent(this.modelName).then(modelBuffer => { // Obtain images in an album. // 1. Create an image picker instance. let photoSelectOptions = new photoAccessHelper.PhotoSelectOptions(); // 2. Set the media file type to IMAGE and set the maximum number of media files that can be selected. photoSelectOptions.MIMEType = photoAccessHelper.PhotoViewMIMETypes.IMAGE_TYPE; // Select images. photoSelectOptions.maxSelectNumber = 1; // Set the maximum number of media files that can be selected. // 3. Create an album picker instance and call select() to open the album page for file selection. After file selection is done, the result set is returned through photoSelectResult. let photoPicker = new photoAccessHelper.PhotoViewPicker(); photoPicker.select(photoSelectOptions, async ( err: BusinessError, photoSelectResult: photoAccessHelper.PhotoSelectResult) => { if (err) { hilog.error(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_ERR: PhotoViewPicker.select failed with err: ${JSON.stringify(err)}`); return; } hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: PhotoViewPicker.select successfully, uri: ${JSON.stringify(photoSelectResult)}`); this.uris = photoSelectResult.photoUris; hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: uri: ${this.uris}`); // Preprocess the image data. try { // 1. Based on the specified URI, call fileIo.openSync to open the file to obtain the FD. let file = fileIo.openSync(this.uris[0], fileIo.OpenMode.READ_ONLY); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: file fd: ${file.fd}`); // 2. Based on the FD, call fileIo.readSync to read the data in the file. let inputBuffer = new ArrayBuffer(4096000); let readLen = fileIo.readSync(file.fd, inputBuffer); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: readSync data to file succeed and inputBuffer size is: ${readLen}`); // 3. Perform image preprocessing through PixelMap. let imageSource = image.createImageSource(file.fd); if (imageSource === undefined) { hilog.error(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_ERR: createImageSource failed.`); return } imageSource.createPixelMap().then((pixelMap) => { pixelMap.getImageInfo().then((info) => { hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: info.width = ${info.size.width}`); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: info.height = ${info.size.height}`); // 4. Crop the image based on the input image size and obtain the image buffer readBuffer. pixelMap.scale(256.0 / info.size.width, 256.0 / info.size.height).then(() => { pixelMap.crop( { x: 16, y: 16, size: { height: this.modelInputHeight, width: this.modelInputWidth } } ).then(async () => { let info = await pixelMap.getImageInfo(); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: crop info.width = ${info.size.width}`); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: crop info.height = ${info.size.height}`); // Set the size of readBuffer. let readBuffer = new ArrayBuffer(this.modelInputHeight * this.modelInputWidth * 4); await pixelMap.readPixelsToBuffer(readBuffer); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: Succeeded in reading image pixel data, buffer: ${readBuffer.byteLength}`); // Convert readBuffer to the float32 format, and standardize the image. const imageArr = new Uint8Array( readBuffer.slice(0, this.modelInputHeight * this.modelInputWidth * 4)); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: imageArr length: ${imageArr.length}`); let means = [0.485, 0.456, 0.406]; let stds = [0.229, 0.224, 0.225]; let float32View = new Float32Array(this.modelInputHeight * this.modelInputWidth * 3); let index = 0; for (let i = 0; i < imageArr.length; i++) { if ((i + 1) % 4 === 0) { float32View[index] = (imageArr[i - 3] / 255.0 - means[0]) / stds[0]; // B float32View[index+1] = (imageArr[i - 2] / 255.0 - means[1]) / stds[1]; // G float32View[index+2] = (imageArr[i - 1] / 255.0 - means[2]) / stds[2]; // R index += 3; } } hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: float32View length: ${float32View.length}`); let printStr = 'float32View data:'; for (let i = 0; i < 20; i++) { printStr += ' ' + float32View[i]; } hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: float32View data: ${printStr}`); let inputs: ArrayBuffer[] = [float32View.buffer]; // predict modelPredict(modelBuffer.buffer.slice(0), inputs).then(outputs => { hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `=========MS_LITE_LOG: MS_LITE predict success=====`); // Print the result. for (let i = 0; i < outputs.length; i++) { let out = new Float32Array(outputs[i].getData()); let printStr = outputs[i].name + ':'; for (let j = 0; j < out.length; j++) { printStr += out[j].toString() + ','; } hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: ${printStr}`); // Obtain the top 5 maximum numbers of categories. this.max = 0; this.maxIndex = 0; this.maxArray = []; this.maxIndexArray = []; let newArray = out.filter(value => value !== this.max) for (let n = 0; n < 5; n++) { this.max = out[0]; this.maxIndex = 0; // Obtain the maximum value. for (let m = 0; m < newArray.length; m++) { if (newArray[m] > this.max) { this.max = newArray[m]; this.maxIndex = m; } } this.maxArray.push(Math.round(this.max * 10000)); this.maxIndexArray.push(this.maxIndex); // Call the array filter function. newArray = newArray.filter(value => value !== this.max) } hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: max: ${this.maxArray}`); hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_LOG: maxIndex: ${this.maxIndexArray}`); } hilog.info(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `=========MS_LITE_LOG END=========`); }) }).catch((error: BusinessError) => { hilog.error(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_ERR: getRawFileContent promise error is: ${error}`); }) }) }) }) // 5. Close the file. fileIo.closeSync(file); } catch (err) { hilog.error(0xFF00, 'MindSporeLiteArkTSDemo', '%{public}s', `MS_LITE_ERR: uri: open file fd failed. ${err}`); } }) }) }) } .width('100%') } .height('100%') } }
Debugging and Verification
-
On DevEco Studio, connect to the device, click Run entry, and build your own HAP.
Launching com.samples.mindsporelitearktsdemo $ hdc shell aa force-stop com.samples.mindsporelitearktsdemo $ hdc shell mkdir data/local/tmp/xxx $ hdc file send C:\Users\xxx\MindSporeLiteArkTSDemo\entry\build\default\outputs\default\entry-default-signed.hap "data/local/tmp/xxx" $ hdc shell bm install -p data/local/tmp/xxx $ hdc shell rm -rf data/local/tmp/xxx $ hdc shell aa start -a EntryAbility -b com.samples.mindsporelitearktsdemo -
Touch the photo button on the device screen, select an image, and touch OK. The classification result of the selected image is displayed on the device screen. In the log printing result, filter images by the keyword MS_LITE. The following information is displayed:
08-06 03:24:33.743 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: PhotoViewPicker.select successfully, photoSelectResult uri: {"photoUris":["file://media/Photo/13/IMG_1501955351_012/plant.jpg"]} 08-06 03:24:33.795 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: readSync data to file succeed and inputBuffer size is:32824 08-06 03:24:34.147 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: crop info.width = 224 08-06 03:24:34.147 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: crop info.height = 224 08-06 03:24:34.160 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: Succeeded in reading image pixel data, buffer: 200704 08-06 03:24:34.970 22547-22547 A03d00/JSAPP com.sampl...liteark+ I =========MS_LITE_LOG: MS_LITE predict start===== 08-06 03:24:35.432 22547-22547 A03d00/JSAPP com.sampl...liteark+ I =========MS_LITE_LOG: MS_LITE predict success===== 08-06 03:24:35.447 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: Default/head-MobileNetV2Head/Sigmoid-op466:0.0000034338463592575863,0.000014028532859811094,9.119685273617506e-7,0.000049100715841632336,9.502661555416125e-7,3.945370394831116e-7,0.04346757382154465,0.00003971960904891603... 08-06 03:24:35.499 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: max:9497,7756,1970,435,46 08-06 03:24:35.499 22547-22547 A03d00/JSAPP com.sampl...liteark+ I MS_LITE_LOG: maxIndex:323,46,13,6,349 08-06 03:24:35.499 22547-22547 A03d00/JSAPP com.sampl...liteark+ I =========MS_LITE_LOG END=========
Effects
Touch the photo button on the device screen, select an image, and touch OK. The top 4 categories of the image are displayed below the image.
Related Samples
The following sample is provided to help you better understand how to develop image classification applications using MindSpore Lite: