BMN

Introduction

Temporal action proposal generation is an challengingand promising task which aims to locate temporal regionsin real-world videos where action or event may occur. Cur-rent bottom-up proposal generation methods can generateproposals with precise boundary, but cannot efficiently gen-erate adequately reliable confidence scores for retrievingproposals. To address these difficulties, we introduce theBoundary-Matching (BM) mechanismto evaluate confi-dence scores of densely distributed proposals, which de-note a proposal as a matching pair of starting and endingboundaries and combine all densely distributed BM pairsinto the BM confidence map. Based on BM mechanism,we propose an effective, efficient and end-to-end proposalgeneration method, namedBoundary-Matching Network(BMN), which generates proposals with precise temporalboundaries as well as reliable confidence scores simultane-ously. The two-branches of BMN are jointly trained in anunified framework. We conduct experiments on two chal-lenging datasets: THUMOS-14 and ActivityNet-1.3, whereBMN shows significant performance improvement with re-markable efficiency and generalizability. Further, combin-ing with existing action classifier, BMN can achieve state-of-the-art temporal action detection performance.

For more details: https://arxiv.org/abs/1907.09702

For original gpu code or potential missing scripts: https://github.com/JJBOY/BMN-Boundary-Matching-Network

Requirements

With PyTorch, you can use pip or conda to install the requirements.

Updated 2022.07: It's recommended to set up environment by cloning from others or installing .whl files.

# for pip
cd $project
pip3.7 install -r requirements.txt
Note: pillow recommends installing a newer version. If the corresponding torchvision version cannot be installed directly, you can use the source code to install the corresponding version. The source code reference link: https://github.com/pytorch/vision，
Suggestion the pillow is 9.1.0 and the torchvision is 0.6.0
CANN 5.0.3

Dataset Preparation

1. For dataset, please refer to BSN open source repository: https://github.com/wzmsltw/BSN-boundary-sensitive-network.pytorch

2. There are two directories named 'data' and 'Evaluation' which can not be uploaded here because of the limit of transmission, please refer to https://github.com/JJBOY/BMN-Boundary-Matching-Network/tree/master, download diretory 'data' and 'Evaluation', then put it under directory 'BMN'.

The structure of this project is illustrated below:

BMN
|--ascend_function             /解决NPU不支持的算子替换函数的目录
|-- test                               /脚本文件夹
|   |--env.sh                          /环境配置文件
|   |--train_full_1p.sh                /单卡精度测试脚本
|   |--train_full_8p.sh                /8卡精度测试脚本
|   |--train_performance_1p.sh         /单卡性能测试脚本
|   |--train_performance_8p.sh         /8卡性能测试脚本
|-- demo.py                            /演示脚本
|-- loss_function.py                   /损失函数脚本
|-- main_8p.py                         /npu主函数，训练启动脚本
|-- models.py                          /模型脚本
|-- opts.py                            /参数脚本
|-- utils.py                           /工具箱脚本
|-- post_proccessing.py                /后处理脚本
|-- eval.py                            /评估脚本
|-- dataset.py                         /数据集预处理脚本

Training

To train a model, run main scripts with the desired model architecture and the path to the ActivityNet dataset:

# training 1p accuracy
bash ./test/train_full_1p.sh --data_path=real_data_path

# training 1p performance
bash ./test/train_performance_1p.sh --data_path=real_data_path

# training 8p accuracy
bash ./test/train_full_8p.sh --data_path=real_data_path

# training 8p performance
bash ./test/train_performance_8p.sh --data_path=real_data_path

# finetuning 1p 
bash test/train_finetune_1p.sh --data_path=real_data_path --pth_path=real_pre_train_model_path

Models are saved in "./checkpoint", and the files generated by models are saved in "./output/result".

BMN Training Result

Acc@100	FPS	Npu_nums	Epochs	AMP_Type
75.11	65	1	10	O1
74.82	553	8	10	O1

BMN PyTorch 1.5 to PyTorch 1.8 Result

Acc@100	FPS	Npu_nums	Epochs	AMP_Type
75.32	62	1	10	O1
75.29	523	8	10	O1

Demo

By running the following instruction, demonstration will be generated and saved in "./output/demo/demo.csv".

python3 demo.py --data_path=real_data_path