Installation Guide

TensorFlow ANNC for Graph Compilation Optimization

Verified Environments

To use the TensorFlow ANNC for graph compilation optimization feature smoothly and securely, ensure that your environment is one of the verified environments.

Hardware Requirements

Table 1 Hardware requirement describes the verified hardware environment.

Table 1 Hardware requirements

Item	Description
CPU	Kunpeng 920 processor (80 cores)

OS Requirements

Table 2 OS describes the verified OS.

Table 2 OS

Item	Version	Description	How to Obtain
OS	openEuler 22.03 LTS SP3	When installing an OS, choose Minimal Install and select Development Tools to minimize manual operations.	Link
Kernel	5.10.0	-	Included in the OS image

Software Requirements

Table 3 Software requirements describes the verified software environments.

Table 3 Software requirements

Item	Version	Description	How to Obtain
Python	3.9.9	Python is an auxiliary tool used during the build of TF Serving. It is used to automatically download dependencies and configure the environment.	Install it using a Yum repository.
CMake	3.22.0	CMake is the build tool of TF Serving. The CMake version must be 3.22.0 or later.	Install it using a Yum repository.
GCC/G++	12.3.1	GNU Compiler Collection (GCC) serves as the compiler for compiling TF Serving into an executable file.	Install it using a Yum repository.
Bazel	6.5.0	Bazel is a powerful build system that enables fast and scalable building. The Bazel version must be 6.4.0 or later.	Link
TensorFlow	2.15.1	TensorFlow, developed by Google, is an open-source machine learning framework. It supports end-to-end AI model development and deployment from research to production.	Install it using a pip repository.

Setting Up the Environment

Prepare the TF Serving compilation environment by following instructions in Configuring the Compilation Environment in the TensorFlow Serving Porting Guide and Table 3 Software requirements.

Compilation and Installation

The backend implementation of TensorFlow ANNC has been merged into the official TensorFlow and TF Serving open-source repositories. Code related to the frontend graph fusion, XLA graph fusion, and operator optimization are available in the ANNC open-source repository hosted on Gitcode, which can be cloned and compiled using git.

Obtaining Code

1. Disable SSL verification for Git.

   git config --global http.sslVerify false
   git config --global https.sslVerify false
   

2. Obtain the TensorFlow and TF Serving code.

   git clone --branch v2.15.0-2509 https://gitcode.com/boostkit/tensorflow.git
   git clone --branch v2.15.1-2509 https://gitcode.com/boostkit/tensorflow-serving.git
   

3. Obtain ANNC code.

   git clone --branch v0.0.2 https://gitcode.com/openeuler/ANNC.git
   

Performing Compilation and Installation

1. Install GCC 12.3.1.

   yum install -y gcc-toolset-12-gcc*
   export PATH=/opt/openEuler/gcc-toolset-12/root/usr/bin/:$PATH
   export LD_LIBRARY_PATH=/opt/openEuler/gcc-toolset-12/root/usr/lib64/:$LD_LIBRARY_PATH
   

2. Compile and install ANNC.

   export ANNC=/path/to/ANNC
   cd $ANNC
   source build.sh
   cp bazel-bin/annc/service/cpu/libannc.so /usr/lib64/
   cp $ANNC/annc/service/cpu/xla/libs/XNNPACK/build/libXNNPACK.so /usr/lib64
   mkdir -p /usr/include/annc 
   cp annc/service/cpu/kdnn_rewriter.h /usr/include/annc/ 
   cp annc/service/cpu/annc_flags.h /usr/include/annc/
   cd python
   python3 setup.py bdist_wheel
   python3 -m pip install dist/*.whl --force-reinstall
   

3. Enable the patch.

   export TF_PATH=/path/to/tensorflow
   export XLA_PATH=/path/to/tensorflow/third_party/xla
   cd $ANNC/install/tfserver/xla
   bash xla2.sh
   

   Figure 1 Successful patching shows an example of successful patching.

   Figure 1 Successful patching

4. Go to the tensorflow-serving directory.

   cd /path/to/tensorflow-serving/
   

5. Create a directory for storing compilation dependencies.

   export DISTDIR=$(pwd)/download
   mkdir -p $DISTDIR
   

6. Set the BAZEL_PATH to the directory that contains the bazel executable file.

   export BAZEL_PATH=/path/to/bazel
   

7. Run the build script to compile the code.

   sh compile_serving.sh --tensorflow_dir /path/to/tensorflow --features gcc12,annc
   

   /path/to/tensorflow specifies the TensorFlow path.

   The build result is the TF Serving binary file tensorflow_model_server, and the file path is /path/to/tensorflow-serving/bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server.

   In the preceding command, gcc12 indicates that GCC 12.3.1 is used for compilation. The compile command in the build script is as follows:

   bazel --output_user_root=$BAZEL_COMPILE_CACHE build -c opt --distdir=$DISTDIR --override_repository=org_tensorflow=$TENSORFLOW_DIR \
   --copt=-march=armv8.3-a+crc --copt=-O3 --copt=-fprefetch-loop-arrays --copt=-Wno-error=maybe-uninitialized  \ 
   --copt=-Werror=stringop-overflow=0 --config=fused_embedding  \ 
   --define tflite_with_xnnpack=false tensorflow_serving/model_servers:tensorflow_model_server
   

   The following describes some parameters:

   • --output_user\root: Bazel compilation cache directory. The default value is /path/to/tensorflow-serving/output. You can set a custom path using the BAZEL_COMPILE_CACHE environment variable. The command is as follows:

     export BAZEL_COMPILE_CACHE=/path/to/your/cache_dir
     

   • --distdir: directory for storing TF Serving compilation dependencies. It ensures reliable access when third-party package download fails.

   • --override_repository: specifies the local TensorFlow build and uses the directory specified by tensorflow_dir as the local TensorFlow.

Common build errors

For details about common build errors and solutions, see FAQs.

TensorFlow Serving Thread Scheduling Optimization

Verified Environments

Hardware Requirements

Table 1 Hardware requirement describes the verified hardware environment.

Table 1 Hardware requirement

Item	Description
CPU	Kunpeng 920 processor (80 cores)

OS Requirements

Table 2 OS describes the verified OS.

Table 2 OS

Item	Version	Description	How to Obtain
OS	openEuler 22.03 LTS SP3	When installing an OS, choose Minimal Install and select Development Tools to minimize manual operations.	Link
Kernel	5.10.0	-	Included in the OS image

Software Requirements

Table 3 Software requirements describes the verified software environments.

Table 3 Software requirements

Item	Version	Description	How to Obtain
Python	3.9.9	Python is an auxiliary tool used during the build of TF Serving. It is used to automatically download dependencies and configure the environment.	Install it using a Yum repository.
CMake	3.22.0	CMake is the build tool of TF Serving. The CMake version must be 3.22.0 or later.	Install it using a Yum repository.
GCC/G++	12.3.1	GNU Compiler Collection (GCC) serves as the compiler for compiling TF Serving into an executable file.	Install it using a Yum repository.
Bazel	6.5.0	Bazel is a powerful build system that enables fast and scalable building. The Bazel version must be 6.4.0 or later.	Link

Environment Preparation

Prepare the TF Serving compilation environment by following instructions in Configuring the Compilation Environment in the TensorFlow Serving Porting Guide.

Compilation and Installation

The patches for TF Serving thread scheduling optimization are now available in the open-source TensorFlow and TF Serving repositories hosted on Gitee. You can clone these repositories via Git and proceed with compilation.

Obtaining Code

1. Disable SSL verification for Git.

   git config --global http.sslVerify false
   git config --global https.sslVerify false
   

2. Pull the code.

   git clone https://gitee.com/openeuler/sra_tensorflow_adapter.git -b v2.15.1.0
   

Performing Compilation

1. Install GCC 12.3.1.

   yum install -y gcc-toolset-12-gcc*
   PATH=/opt/openEuler/gcc-toolset-12/root/usr/bin/:$PATH
   LD_LIBRARY_PATH=/opt/openEuler/gcc-toolset-12/root/usr/lib64/:$LD_LIBRARY_PATH
   

2. Go to the sra_tensorflow_adapter directory.

   cd sra_tensorflow_adapter/
   

3. Create a directory for storing compilation dependencies. The path is /path/to/sra_tensorflow_adapter/serving/download.

   export DISTDIR=$(pwd)/serving/download
   mkdir -p $DISTDIR
   

4. Set the BAZEL_PATH to the directory that contains the bazel executable file.

   export BAZEL_PATH=/path/to/bazel
   

5. Run the build script to compile the code.

   sh compile_serving.sh gcc12
   

   The build result is the TF Serving binary file tensorflow_model_server, and the file path is /path/to/sra_tensorflow_adapter/serving/bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server.

   In the preceding command, gcc12 indicates that GCC 12.3.1 is used for compilation. The compile command in the build script is as follows:

   bazel --output_user_root=$BAZEL_COMPILE_CACHE build -c opt --distdir=$DISTDIR --override_repository=org_tensorflow=$TENSORFLOW_DIR --copt=-march=armv8.3-a+crc --copt=-O3 --copt=-fprefetch-loop-arrays --copt=-Wno-error=maybe-uninitialized --copt=-Werror=stringop-overflow=0 tensorflow_serving/model_servers:tensorflow_model_server
   

   The following describes some parameters:

   • output_user_root: Bazel compilation cache directory. The default value is /path/to/sra_tensorflow_adapter/serving/output. You can set a custom path using the BAZEL_COMPILE_CACHE environment variable. The command is as follows:

     export BAZEL_COMPILE_CACHE=/path/to/your/cache_dir
     

   • distdir: directory for storing TF Serving compilation dependencies. It ensures reliable access when third-party package download fails.

   • override_repository: specifies the local TensorFlow build. The /path/to/sra_tensorflow_adapter/tensorflow directory is automatically identified as the build dependency.

Common build errors

For details about common build errors and solutions, see FAQs.

TensorFlow–KDNN Integration

Kunpeng Deep Neural Network Library (KDNN) is a high-performance AI operator library optimized for the Kunpeng platform. These optimizations are delivered by integrating operators such as MatMul, FusedMatMul, and SparseMatmul into TensorFlow. Integrating KDNN can reduce the latency of Neural Network (NN) operators and greatly improve the model inference performance.

1. Obtain the KDNN software package for GCC. Decompress the ZIP file to obtain the RPM installation package.

2. Install KDNN.

   rpm -ivh boostcore-kdnn-3.0.0-1.aarch64.rpm
   

   The header file installation directory is /usr/local/kdnn/include, and the library file installation directories are /usr/local/kdnn/lib/threadpool and /usr/local/kdnn/lib/omp.

3. Install KDNN header files and static libraries to the /path/to/tensorflow/third_party/KDNN directory.

   export TF_PATH=/path/to/tensorflow
   mkdir -p $TF_PATH/third_party/KDNN/src
   cp -r /usr/local/kdnn/include $TF_PATH/third_party/KDNN
   cp -r /usr/local/kdnn/lib/threadpool/libkdnn.a $TF_PATH/third_party/KDNN/src
   

4. Go to the KDNN directory and apply the header file patch to fix TensorFlow's exception handling limitation.

   cd $TF_PATH/third_party/KDNN
   patch -p0 < $TF_PATH/third_party/KDNN/tensorflow_kdnn_include_adapter.patch
   

5. Run the build script to compile the code.

   cd /path/to/serving
   sh compile_serving.sh --tensorflow_dir /path/to/tensorflow --features gcc12,kdnn
   

FAQs About Installation

• If the message "unable to find valid certification path to requested target" is displayed, you can fix it by following the instructions in Failed to Verify the Certificate When Compiling TensorFlow 2.13.0 Source Code in the TensorFlow Porting Guide.

• If the message "Error in download_and_extract" is displayed, you can fix it by following the instructions in Failed to Download the TF-Serving Source Code Dependency in the TensorFlow Serving Porting Guide. The directory specified by --distdir is /path/to/tensorflow-serving/download.

• If the system displays a message indicating that the org_boost dependency library fails to be pulled, you can fix it by following the instructions in Failed to Obtain the Dependency of the org_boost Sub-repository in the TensorFlow Serving Porting Guide.

• If the system displays a message indicating that the Golang website certificate is unavailable, you can fix it by following the instructions in No Golang Website Certificate in the TensorFlow Serving Porting Guide.

• If the system displays a message indicating that the Golang download fails during ANNC compilation, you can fix it by following the instructions in Failed to Download the TF-Serving Source Code Dependency in the TensorFlow Serving Porting Guide. The directory specified by --distdir can be specified in the build.sh script.

• If the system displays a message indicating that the upb.c file fails to be compiled, you can fix it by following the instructions in Syntax Error Reported During upb.c Compilation in the TensorFlow Serving Porting Guide.