OpenGL Soft Decoding Rendering Demo (Soft Surface Demo)

Overview

This sample demonstrates how to implement software video decoding (CPU decoding) in a flutter application and manually push decoded frame data to Flutter's Texture widget for rendering via the NativeWindow BufferQueue.

Unlike the traditional GPU hardware decoding + direct Surface rendering approach, this sample uses Buffer mode to decode video, performs NV12 → RGBA color space conversion on the CPU, then writes RGBA data into the NativeWindow buffer via mmap. Flutter Engine's onFrameAvailable callback is triggered to refresh the UI. This approach is suitable for devices with limited GPU capabilities or scenarios requiring additional CPU-side processing of decoded frames (e.g., filters, watermarks, analysis).

Architecture

┌──────────────────────────────────────────────────────┐
│                    Flutter (Dart)                     │
│                                                      │
│  SoftSurfacePage                                     │
│  ├─ MethodChannel('com.example.softsurface/render')  │
│  ├─ createTexture()  → Create Texture widget         │
│  ├─ startRendering() → Start decoding & rendering    │
│  └─ stopRendering()  → Stop decoding & rendering     │
│                  │                                   │
│          Texture(textureId)                          │
│          (640×360 AspectRatio)                       │
└──────────────────┬───────────────────────────────────┘
                   │ MethodChannel
┌──────────────────▼───────────────────────────────────┐
│                Plugin Layer (ArkTS/ETS)               │
│                                                      │
│  SoftSurfacePlugin                                   │
│  ├─ createTexture: Register texture to TextureRegistry│
│  ├─ startRendering: Open rawfile video → call C++    │
│  └─ stopRendering:  Call C++ to stop                 │
└──────────────────┬───────────────────────────────────┘
                   │ NAPI (libsoftrender.so)
┌──────────────────▼───────────────────────────────────┐
│                Native Layer (C++)                     │
│                                                      │
│  soft_render.cpp                                     │
│  ├─ OH_AVSource: Create media source from rawfile FD │
│  ├─ OH_AVDemuxer: Demux, extract video track        │
│  ├─ OH_VideoDecoder: Buffer mode decode (no Surface) │
│  ├─ NV12ToRGBA(): CPU color space conversion         │
│  ├─ mmap → Write to NativeWindow Buffer → FlushBuffer│
│  └─ Auto SeekToTime(0) on EOS for loop playback      │
└──────────────────────────────────────────────────────┘

Data Flow

video1.mp4 (rawfile)
    │
    ▼
OH_AVSource (FD)
    │
    ▼
OH_AVDemuxer (extract H.264/H.265 stream packets)
    │
    ▼
OH_VideoDecoder (Buffer mode decode → NV12 raw frames)
    │
    ▼
NV12ToRGBA() (CPU color space conversion)
    │
    ▼
mmap(NativeWindow Buffer) → Write RGBA pixels
    │
    ▼
OH_NativeWindow_NativeWindowFlushBuffer
    │
    ▼
Flutter Engine onFrameAvailable → Texture widget refresh

Project Structure

opengl_for_soft_decoding/
├── lib/
│   └── main.dart                          # Flutter entry point, UI & MethodChannel communication
├── ohos/
│   ├── AppScope/
│   │   └── app.json5                      # App configuration (com.example.softsurface)
│   ├── entry/
│   │   └── src/main/
│   │       ├── cpp/
│   │       │   ├── CMakeLists.txt         # CMake build config, links multimedia & NativeWindow libs
│   │       │   └── soft_render.cpp        # Core decode & render logic (NAPI module)
│   │       ├── ets/
│   │       │   ├── entryability/
│   │       │   │   └── EntryAbility.ets   # App entry Ability, registers plugins
│   │       │   ├── pages/
│   │       │   │   └── Index.ets          # Main page, hosts FlutterPage
│   │       │   └── plugins/
│   │       │       ├── SoftSurfacePlugin.ets        # Texture creation & render control plugin
│   │       │       └── GeneratedPluginRegistrant.ets
│   │       ├── module.json5               # Module config (permission: INTERNET)
│   │       └── resources/
│   │           └── rawfile/
│   │               ├── video1.mp4         # Sample video file
│   │               └── framesconfig.json  # Dynamic frame rate control config
│   └── oh-package.json5
├── pubspec.yaml                           # Flutter dependency configuration
└── test/
    └── widget_test.dart

How to Use

Build & Run

1. Navigate to the project directory:

   cd opengl_for_soft_decoding
   

2. Install Flutter dependencies:

   flutter pub get
   

3. Open the ohos/ directory in DevEco Studio and ensure the C++ native library compiles successfully (libsoftrender.so).

4. Connect device or launch an emulator, then run the app.

Steps

After the app launches, the UI consists of two buttons and a video rendering area:

Step	Action	Description
1	Tap "Create Texture" button	Registers a new texture with the Flutter TextureRegistry, returns a textureId, and displays a black rendering area (640×360 aspect ratio)
2	Tap "Start" button	Starts software video decoding and rendering. The plugin opens rawfile/video1.mp4, passes the file descriptor and NativeWindow pointer to the C++ layer, and starts the decode loop. The status bar shows "Rendering..."
3	Tap "Stop" button	Stops decoding and releases all resources (decoder, demuxer, media source, etc.). The status bar shows "Stopped"

Note: The video automatically loops back to the beginning upon reaching the end (SeekToTime(0)).

Replacing the Video File

Place your custom .mp4 video file at the following path, named video1.mp4 (or modify the filename in SoftSurfacePlugin.ets):

ohos/entry/src/main/resources/rawfile/video1.mp4

Use Cases

• CPU-side post-processing of decoded frames (image analysis, filters, watermark overlay, etc.)
• Target devices with limited GPU decoding capabilities or lacking support for specific codecs
• Scenarios requiring precise control over the timing and content of each rendered frame