Sample Usage Guide

1. Function Description

This sample uses BatchNorm operator's optional inputs for graph construction, aiming to help graph construction developers quickly understand the definition of optional inputs and how to use this type of operator for graph construction.

2. Directory Structure

cpp/
├── src/
|   └── CMakeLists.txt            // CMake build file
|   └── es_showcase.h             // Header file
|   └── make_batchnorm_graph.cpp  // Sample file
├── CMakeLists.txt                // CMake build file
├── main.cpp                      // Program main entry
├── README.md                     // README file
├── run_sample.sh                 // Execution script
├── utils.h                       // Utility file

3. Usage

3.1 Prepare CANN Package

• Correctly install toolkit and ops packages following the installation guide Environment Preparation
• Set environment variables (assuming the package is installed in /usr/local/Ascend/)

source /usr/local/Ascend/cann/set_env.sh

3.2 Build and Execute

1.2.1 Generate ES Interface and Build Graph for DUMP

Simply run the following command to clean up, generate interface, build graph and DUMP graph:

bash run_sample.sh

The current behavior of run_sample.sh is: automatically clean up old build, build sample and default execute sample dump. When you see the following message, it means successful execution:

[Success] sample executed successfully, pbtxt dump has been generated in current directory. This file starts with ge_onnx_ and can be opened in netron for display

1.2.2 Output File Description

After successful execution, the following file will be generated in current directory:

ge_onnx_*.pbtxt - Protobuf text format of graph structure, can be viewed with netron

1.2.3 Build Graph and Execute

Besides basic graph building and dump functionality, esb_sample supports building graph and actually executing computation.

bash run_sample.sh -t sample_and_run

This command will:

1. Automatically generate ES interface
2. Compile sample program
3. Generate dump graph, run graph and output computation results

After successful execution, you will see:

[Success] sample_and_run executed successfully, pbtxt and data output dump have been generated in current directory

You can check computation results through data file

3.3 Log Printing

If log printing is needed during executable program execution to assist debugging, set the following environment variables before bash run_sample.sh to print logs to screen:

export ASCEND_SLOG_PRINT_TO_STDOUT=1 #Print logs to screen
export ASCEND_GLOBAL_LOG_LEVEL=0 #Log level set to debug level

1.4 Graph Compilation DUMP

If DUMP graph is needed during executable program execution to assist graph compilation debugging, set the following environment variables before bash run_sample.sh -t sample_and_run to DUMP graph to execution path:

export DUMP_GE_GRAPH=2

4. Core Concept Introduction

4.1 Graph Construction Steps

• Create graph builder (to provide context, workspace and build-related methods needed for graph construction)
• Add starting nodes (starting nodes refer to nodes without input dependencies, usually including graph inputs (like Data nodes) and weight constants (like Const nodes))
• Add intermediate nodes (intermediate nodes are computation nodes with input dependencies, usually generated by user graph construction logic, and connected using existing nodes as inputs)
• Set graph output (explicitly specify graph output nodes as endpoints of computation results)

4.2 Concept Description

Optional input refers to certain inputs of an operator that are non-mandatory inputs.

For example, BatchNorm operator prototype is shown below, ES graph construction generated API is BatchNorm(), supporting use at C++ layer

  REG_OP(BatchNorm)
    .INPUT(x, TensorType({DT_FLOAT16,DT_FLOAT}))
    .INPUT(scale, TensorType({DT_FLOAT}))
    .INPUT(offset, TensorType({DT_FLOAT}))
    .OPTIONAL_INPUT(mean, TensorType({DT_FLOAT}))
    .OPTIONAL_INPUT(variance, TensorType({DT_FLOAT}))
    .OUTPUT(y, TensorType({DT_FLOAT16,DT_FLOAT}))
    .OUTPUT(batch_mean, TensorType({DT_FLOAT}))
    .OUTPUT(batch_variance, TensorType({DT_FLOAT}))
    .OUTPUT(reserve_space_1, TensorType({DT_FLOAT}))
    .OUTPUT(reserve_space_2, TensorType({DT_FLOAT}))
    .OUTPUT(reserve_space_3, TensorType({DT_FLOAT}))
    .ATTR(epsilon, Float, 0.0001f)
    .ATTR(data_format, String, "NHWC")
    .ATTR(is_training, Bool, true)
    .ATTR(exponential_avg_factor, Float, 1.0)
    .OP_END_FACTORY_REG(BatchNorm)

Its corresponding function prototype is:

• Function name: BatchNorm
• Parameters: Total 9, sequentially x, scale, offset, mean (optional input), variance (optional input), epsilon, data_format, is_training, exponential_avg_factor
• Return values: Outputs y, batch_mean, batch_variance, reserve_space_1, reserve_space_2, reserve_space_3

In C API:

EsBatchNormOutput EsBatchNorm(EsCTensorHolder *x, EsCTensorHolder *scale, EsCTensorHolder *offset, EsCTensorHolder *mean, EsCTensorHolder *variance, float epsilon, const char *data_format, bool is_training, float exponential_avg_factor);
typedef struct {
    EsCTensorHolder *y;
    EsCTensorHolder *batch_mean;
    EsCTensorHolder *batch_variance;
    EsCTensorHolder *reserve_space_1;
    EsCTensorHolder *reserve_space_2;
    EsCTensorHolder *reserve_space_3;
} EsBatchNormOutput;

In C++ API:

BatchNormOutput BatchNorm(const EsTensorLike &x, const EsTensorLike &scale, const EsTensorLike &offset, const EsTensorLike &mean=nullptr, const EsTensorLike &variance=nullptr, float epsilon=0.000100, const char *data_format="NHWC", bool is_training=true, float exponential_avg_factor=1.000000);
struct BatchNormOutput {
    EsTensorHolder y;
    EsTensorHolder batch_mean;
    EsTensorHolder batch_variance;
    EsTensorHolder reserve_space_1;
    EsTensorHolder reserve_space_2;
    EsTensorHolder reserve_space_3;
};

Note: Use TensorLike type to express input, to support cases where actual parameters can directly pass numeric values