component_drivers:Hardware component drivers framework with unified management interface

Hardware component drivers framework with unified management interface

Branch444Tags189
FilesLast commitLast update
2 months ago
1 month ago
11 months ago
14 days ago
3 days ago
3 days ago
7 hours ago
2 days ago
1 month ago
13 days ago
7 hours ago
14 days ago
4 months ago
4 months ago
1 month ago
1 day ago
2 months ago
1 month ago
4 months ago
3 months ago
1 year ago
2 months ago
14 days ago
7 hours ago
6 days ago
4 months ago
14 days ago
23 days ago
13 days ago
16 days ago
1 month ago

component_drivers

Verification Notice: This document serves as the sole authoritative guide for building and testing the component_drivers component. The verification process must be executed in strict accordance with this guide. Documents in the docs/ directory are feature specifications only and do not cover the build process. This component supports Ubuntu 24.04 LTS (x86-64) and openEuler 24.03 LTS (aarch64/Kunpeng) builds, with Ubuntu 24.04 LTS recommended. Other systems such as CentOS and Debian are not supported; do not attempt to use alternative images.

component_drivers is the component driver component of openUBMC, providing a foundational hardware device driver framework and device-specific adapter implementations (such as PCIe NIC, GPU, PSU, and VRD drivers) for monitoring and managing various hardware devices.


System Requirements

Item Requirement
Operating System Ubuntu 24.04 LTS (x86-64, recommended)
openEuler 24.03 LTS (aarch64/Kunpeng, verified baseline LTS-SP4)
Architecture x86-64 (amd64), aarch64 (arm64)
Python >= 3.10 (required by init.py)

Build tools such as bingo and Conan are automatically installed by init.py — no manual preparation needed.

Container Build Images

Docker containers are recommended for building. Use the following images to quickly start a development environment.

docker pull swr.cn-north-4.myhuaweicloud.com/openubmc/ubuntu:24.04.2_26.06

aarch64 (openEuler, ARM build)

docker pull --platform linux/arm64 swr.cn-north-4.myhuaweicloud.com/openubmc/openeuler:20261024_1

Build

Build from Source Code

Three build modes are available, all supporting ccache acceleration and libmcpp precompilation cache.

Mode Command Feature
smart_build.sh One-line command Automated workflow: configuration → compilation → test. The number of concurrent tasks is automatically optimized.
setup.sh Step-by-step execution Environment configuration only. Compilation and testing must be run manually, making this mode ideal for step-by-step debugging.
run_tests_loop.sh Loop + GDB Multiple rounds of stability/random reproduction. Each round can recompile based on the specified cleanup level, and logs are stored in test_failure_logs/.
Traditional Meson Manual command No scripts required. libmcpp is compiled as a subproject (slower).

This method provides a one-click build workflow with automated ccache, libmcpp precompilation cache, and concurrency optimization. The parameters match --help in scripts/smart_build.sh.

./scripts/smart_build.sh                   # Standard build (auto-detects and reuses cache: configuration → compilation → test)
./scripts/smart_build.sh --clean|--clean-build|-c   # Level-1 cleanup: deletes and rebuilds builddir only (retains libmcpp precompilation cache)
./scripts/smart_build.sh --clean-prebuilt|-C        # Level-2 cleanup: performs Level-1 cleanup + removes libmcpp precompiled artifacts (e.g., .libmcpp_prebuilt)
./scripts/smart_build.sh --clean-all|-X             # Level-3 cleanup: performs Level-2 cleanup + deletes subprojects/libmcpp + purges ccache
./scripts/smart_build.sh --purge-cache              # Equivalent to --clean-all.
./scripts/smart_build.sh --setup-only             # Configures the environment only, skipping compilation
./scripts/smart_build.sh --no-test                # Compiles code but skips tests.
./scripts/smart_build.sh --test-timeout-multiplier 2  # Test timeout multiplier (default: 1)
./scripts/smart_build.sh --coverage               # Enables code coverage.
./scripts/smart_build.sh --fast-coverage          # Enables fast coverage mode (implies --coverage).
./scripts/smart_build.sh --coverage-output <dir>    # Coverage output directory (default: coverage)
./scripts/smart_build.sh -Dtests=false            # Passes additional Meson parameters (multiple -D... allowed)
./scripts/smart_build.sh -h|--help                # Displays the full help.

Method 2: setup.sh (Step-by-Step Execution)

This method shares the same underlying mechanics as smart_build.sh (ccache wrapper + libmcpp precompilation + pkg-config), but it only handles the initial configuration. You must run compilation and testing tasks manually.

# 1. Configure the environment. (The libmcpp will be automatically compiled and cached during the first configuration.)
./setup.sh                     # Standard configuration (reuses cache)
./setup.sh --clean             # Clears the libmcpp cache and reconfigures it.
./setup.sh -Dtests=false       # Passes additional Meson parameters.

# 2. Compile. (setup.sh generates the build.sh auxiliary script.)
./build.sh                     # or meson compile -C builddir -j $(nproc)

# 3. Run the test.
meson test -C builddir

Method 3: Traditional Meson Manual Build

This method does not use any script. libmcpp is compiled as a Meson subproject using the .wrap file, which requires many steps and increases compilation time.

Note: If the CCACHE_DIR of the compiler points to the read-only partition, fix this issue first. Compilation will fail otherwise.

# Fix the ccache read-only issue (required for each new terminal).
export CCACHE_DIR=~/.ccache && mkdir -p $CCACHE_DIR

# Configuration + compilation + testing
rm -rf subprojects/libmcpp builddir
meson setup builddir
((n=$(nproc)/4)); meson compile -C builddir -j $n
meson test -C builddir

# Single-module compilation and testing
meson compile -C builddir tests/drivers/chip/lm75/test_lm75_abi
meson test -C builddir --suite=lm75

Loop stability test: run_tests_loop.sh

This script runs tests in multiple rounds of loops (via GDB) to reproduce occasional failures and perform stability stress testing. In every loop iteration, smart_build.sh <Clearing_level> --no-test is called (using the same cleanup levels as smart_build.sh, but bypassing the built-in meson test). Then, this script uses GDB to execute the test. When timeout or blocking occurs, the script captures the thread backtrace and logs it to the test_failure_logs/ directory at the project root (consistent with the definition in the script).

Command line options (consistent with the descriptions in the scripts/run_tests_loop.sh header)

Option Description
-n, --no-build Skips the build phase and directly run tests (requires an existing builddir).
-t, --timeout SEC Timeout interval of a single test (in seconds; default: 300)
-i, --iterations N Loop count (default: 100)
-j, --jobs N Parallel compilation degree (default value: 0 for auto-calculation). When smart_build is used for build, the internal concurrency is still calculated. This parameter is used as a prompt or to overwrite BUILD_JOBS.
-f, --filter FILTER gtest filter, for example, "-ModuleManagerTest.*:-mMicroComponentTest*".
-c, --clean, --clean-build Level-1 cleanup, which is passed to smart_build (default. This parameter can also be specified by the environment variable RUN_TESTS_SMART_CLEAN if -c/-C/-X is not specified.).
-C, --clean-prebuilt Level-2 cleanup (Level-1 cleanup + cleanup of libmcpp precompilation products)
-X, --clean-all, --purge-cache Level-3 cleanup (Level-2 cleanup + deletion of subprojects/libmcpp + clearing of ccache)
-h, --help Shows help information.

Environment variables (can be used together with command lines): MAX_ITERATIONS, TEST_TIMEOUT, BUILD_JOBS, GTEST_FILTER, and RUN_TESTS_SMART_CLEAN. The value is -c, -C, or -X. If no command line is specified for cleanup, the default value is used.

# Default: Level-1 cleanup + smart_build --no-test in each round; then run GDB test; run 100 iterations with 300s timeout
./scripts/run_tests_loop.sh

# Reuse an existing `builddir` for repeated tests only (code not modified, occasionally reproduced).
./scripts/run_tests_loop.sh --no-build

# Exclude unstable test cases and perform stress tests on other test cases.
./scripts/run_tests_loop.sh -f "-ModuleManagerTest.*:-mMicroComponentTest*"

# A large number of loops for a single case (build skipped in this example)
./scripts/run_tests_loop.sh -n -i 500 -f "EepromAbiTest.ReadWrite"

# Extend the timeout period and reduce the iterations count.
./scripts/run_tests_loop.sh --timeout 600 --iterations 10

# Limit the parallelism degree on machines with low memory.
./scripts/run_tests_loop.sh --jobs 2

# CI: environment variables + skipping build
TEST_TIMEOUT=600 MAX_ITERATIONS=50 GTEST_FILTER="SensorTest.*" \
  ./scripts/run_tests_loop.sh --no-build

Compilation Duration Analysis (with Timestamp Logs)

Use ts (from moreutils and apt-get install -y moreutils) to add a relative timestamp to each line of bingo build output to locate time-consuming bottlenecks

# Relative timestamp + complete stdout/stderr + total real/user/sys; The output file contains the date for before/after comparison.
{ time bingo build; } 2>&1 | ts -s '[%H:%M:%.S]' | tee tests/bingo-build-$(date +%Y%m%d-%H%M%S).log
{ time bingo build -jit -bt release; } 2>&1 | ts -s '[%H:%M:%.S]' | tee tests/bingo-build-$(date +%Y%m%d-%H%M%S).log
{ time bingo test; } 2>&1 | ts -s '[%H:%M:%.S]' | tee tests/bingo-test-$(date +%Y%m%d-%H%M%S).log


  • { time ...; } 2>&1 enables time statistics and bingo stderr to enter the pipe (otherwise, they will not be written into the file).
  • ts -s '[%H:%M:%.S]' records relative timestamps at millisecond precision since command execution start so that you can directly see what is being done at a specific minute when reading logs.
  • To use the absolute wall clock time, remove -s: ts '[%F %T]'.
  • 2>&1 must be included. Otherwise, the errors and progress of conan/meson/ninja in stderr will not be logged.

Example log format:

[00:00:00.012] Running: conan create ...
[00:00:45.321] ======== Installing packages ========
[00:03:12.880] [1/528] Compiling C++ object ...
[00:08:15.441] [528/528] Linking target ...
real    8m16.123s
user    54m12.000s
sys     2m31.200s

After obtaining the logs, you can use grep to quickly extract the time consumed by key nodes:

grep -E "(Calling source|Calling build|Calling package|meson setup|\[[0-9]+/[0-9]+\].*(Compiling|Linking))" \
  temp/bingo-build-*.log | head -40

The same rule applies to bingo test:

{ time bingo test; } 2>&1 | ts -s '[%H:%M:%.S]' | tee temp/bingo-test-$(date +%Y%m%d-%H%M%S).log

CI / bingo for Building Environment Variable Switches

bingo build / bingo test uses the Conan + Meson driver chain, shares meson.build with smart_build.sh, but is independent of the scripts in scripts/. The following switches can be directly added before the command or exported to the CI environment.

Variable Default Purpose When to Use
CONAN_UNITY 1 (on) =0: disables the Unity Build (conanfile.py::generate()) of the main project. Locate ODR/anonymous namespace conflicts caused by Unity merging. Retain the default value when Unity is normal.
CONAN_UNITY_SIZE 8 Adjusts the number of translation units merged in each unity source file. If you want to further reduce the number of compilation steps, try the value 12. Before increasing the value, ensure that there is no anonymous namespace / MC_REGISTER_* conflict in each target.
TEST_TIMEOUT 30 (seconds) Timeout of each executable file in bingo test (test/unit/test.lua) Increase the value to 60 for slow machines (such as KVM). When adding a long-time test, preferentially increase the value of ADD_TEST_TIMEOUT locally instead of globally.
INCLUDE_HANG_TESTS Not set (excluded) =1: restores the running of 10 known suspended PCIe NIC/GPU tests. Fix the fork/dlopen/harbor dependencies of these tests and run them locally.

Example:

# Standard CI: Unity ON + timeout 30s + skip hang test by default
bingo build
bingo test

# Temporarily disable Unity to locate conflicts.
CONAN_UNITY=0 bingo build

# Put them back when fixing the suspension test locally.
INCLUDE_HANG_TESTS=1 TEST_TIMEOUT=60 bingo test

# Aggressively compress Unity (conflicts need to be verified target by target).
CONAN_UNITY_SIZE=12 bingo build

Current suspension whitelist (see test/unit/test.lua; included in excluded_executables by default):

libmcpp_test                          -- harbor/IPC limit
test_awm_m11p_abi                     -- innosilicon GPU ABI
test_boardcom_interface               -- boardcom NIC interface
test_cx4_card / test_cx4_om / test_cx4_port     -- Mellanox CX4
test_cx5_card / test_cx5_om / test_cx5_port     -- Mellanox CX5
test_cx6_card / test_cx6_om / test_cx6_port     -- Mellanox CX6

These tests are suspended for 120s in the CI docker due to the linker lock overhead of fork + dlopen 46.so. After this issue is resolved, the execution time for bingo test to run the remaining 82 tests drops to 26 minutes and 55 seconds, compared to 42 minutes when the process timed out or failed to complete. For details, see [docs/Q2. Compilation Time Optimization.md §0.10](docs/Q2. Compilation Time Optimization.md).


Compilation Speed Comparison

Metric Traditional Approach smart_build (First Time) smart_build + ccache Hot Cache
ninja steps 1879 1705 1705
libmcpp compilation 166 steps 0 steps 0 steps
Full compilation time ~30min ~23min ~52s
Traditional approach:
  component_drivers → libmcpp compilation (166 steps) → compilation of its own code (1435 steps)
                      Recompilation is required each time rm -rf builddir is executed.

smart_build / setup.sh:
  First time: 173 steps for libmcpp compilation → Cache to .libmcpp_prebuilt/
  Subsequent: Directly reuse the cache → Compile only its own code (1435 steps), with 0 steps for libmcpp
  ccache hot cache: The compilation can still be completed within 52 seconds after builddir is rebuilt.

FAQ

ccache: error: Read-only file system

The CCACHE_DIR=/opt/cache/ccache of the compilation machine points to the read-only partition. smart_build.sh / setup.sh has been fixed. If you use the traditional approach, you need to manually run the following command:

export CCACHE_DIR=~/.ccache && mkdir -p $CCACHE_DIR

System Suspension During Compilation (OOM)

When 128 cores are fully concurrent, each g++ occupies 1 to 2 GB of memory. smart_build.sh automatically optimizes the number of concurrent tasks. For the traditional approach, run the following command to set a limit:

((n=$(nproc)/4)); meson compile -C builddir -j $n

All ABI Test Failure (cannot open shared object file)

The _abi test loads the driver using dlopen("./opt/bmc/drivers/libXxx.so"). If builddir/opt/bmc/drivers/ is empty, all ABI tests will fail. smart_build.sh and build.sh have built-in automatic verification. In traditional mode, ensure complete compilation:

ninja -C builddir         # Ensure that all copy_lib* targets are executed.
meson test -C builddir    # Run the test again.

Rebuild After the libmcpp Source Code Is Updated

./scripts/smart_build.sh --clean-all   # or -X / --purge-cache; or
./setup.sh --clean                     # or
rm -rf .libmcpp_prebuilt subprojects/libmcpp builddir && meson setup builddir

Introduction

Hardware component drivers framework with unified management interface

Customize your domain