已关闭

[Bug]: BEVFormer模型训练拉不起来 #358

feiyangR创建于 25 天前关闭于 22 天前

feiyangR

25 天前创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

API 令牌或密钥
密码或身份验证凭证
私有网址或接口地址
个人或机密数据
...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

芯片类型: Ascend950DT
操作系统:  openEuler 24.03 (LTS-SP3)

HDK: 25.6.rc1
cann: 9.1.0

torch: 2.7.1+cpu
torch_npu: 2.7.1.post6

TorchVision: 0.22.1
OpenCV: 4.11.0
MMCV: 1.7.2
MMCV Compiler: GCC 12.3
MMCV CUDA Compiler: not available
MMDetection: 2.24.0
MMSegmentation: 0.30.0
MMDetection3D: 1.0.0rc4+66b65f3
spconv2.0: False
例如：
- 操作系统
- 昇腾硬件信息
- CANN软件版本
- 安装的对应软件版本

🐛 问题描述

bash test/train_full_8p_base_fp32.sh --epochs=4 --batch-size=1 --num-npu=8
执行后训练拉起失败 , 且打屏日志没有明显报错 , plog也没有报错。

以下是部分训练打屏日志(最后两百行)

def backward(ctx, grad_output: torch.Tensor) -> tuple:
INFO:torch_npu.env:get env ASCEND_CUSTOM_OPP_PATH = /opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/packages/vendors/customize_arch35
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:26: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=CAST_INPUT)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:84: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_out):
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:20: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=torch.float32)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:50: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_output: torch.Tensor) -> tuple:
INFO:torch_npu.env:get env ASCEND_CUSTOM_OPP_PATH = /opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/packages/vendors/customize_arch35
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmcv/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmdetection3d/mmdet3d/core/evaluation/kitti_utils/eval.py:10: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
WARNING:trimesh:No FCL -- collision checking will not work
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:26: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=CAST_INPUT)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:84: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_out):
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:20: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=torch.float32)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:50: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_output: torch.Tensor) -> tuple:
INFO:torch_npu.env:get env ASCEND_CUSTOM_OPP_PATH = /opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/packages/vendors/customize_arch35
INFO:torch_npu.env:get env TERM = xterm
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmdetection3d/mmdet3d/core/evaluation/kitti_utils/eval.py:10: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmcv/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
WARNING:trimesh:No FCL -- collision checking will not work
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmcv/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmcv/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
warnings.warn(
INFO:torch_npu.env:get env TERM = xterm
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmdetection3d/mmdet3d/core/evaluation/kitti_utils/eval.py:10: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:26: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=CAST_INPUT)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:84: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_out):
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:20: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=torch.float32)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:50: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_output: torch.Tensor) -> tuple:
INFO:torch_npu.env:get env ASCEND_CUSTOM_OPP_PATH = /opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/packages/vendors/customize_arch35
WARNING:trimesh:No FCL -- collision checking will not work
/home/z30085913/DrivingSDK/model_examples/BEVFormer/mmdetection3d/mmdet3d/core/evaluation/kitti_utils/eval.py:10: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
WARNING:trimesh:No FCL -- collision checking will not work
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:26: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=CAST_INPUT)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:26: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=CAST_INPUT)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:84: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_out):
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/modulated_deform_conv2d.py:84: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_out):
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:20: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=torch.float32)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:20: FutureWarning: torch_npu.npu.amp.custom_fwd(args...) is deprecated. Please use torch.amp.custom_fwd(args..., device_type='npu') instead.
@custom_fwd(cast_inputs=torch.float32)
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:50: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_output: torch.Tensor) -> tuple:
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/ops/multi_scale_deformable_attn.py:50: FutureWarning: torch_npu.npu.amp.custom_bwd(args...) is deprecated. Please use torch.amp.custom_bwd(args..., device_type='npu') instead.
def backward(ctx, grad_output: torch.Tensor) -> tuple:
INFO:torch_npu.env:get env ASCEND_CUSTOM_OPP_PATH = /opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/packages/vendors/customize_arch35
INFO:torch_npu.env:get env ASCEND_CUSTOM_OPP_PATH = /opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/mx_driving/packages/vendors/customize_arch35
/opt/conda/envs/torch2.7.1/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:360: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.load, torch.set_default_device, torch.get_device_module, torch.sparse_compressed_tensor, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.Tensor.pin_memory, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************