/label add triaged
export ASCEND_LAUNCH_BLOCKING=1 我设置了这个环境变量后
日志报错又变成这个了
Thread 0x0000fffd4ff4f120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_innerFatal Python error:
PyThreadState_Get File : "the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py
"Python runtime state: , line initialized1002
in
_bootstrapThread 0x
0000fffd3ffff120
(most recent call first):
Thread 0x File 0000fffd4e71f120" (most recent call first):
/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py File ""/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line , line 327327 in in waitwait
File File ""/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py"", line , line 231231 in in _feed_feed
File File ""/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py""Fatal Python error: , line , line PyThreadState_Get982982: in in the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)runrun
Python runtime state: File File initialized""
/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py
""Thread 0x, line , line Fatal Python error: 0000fffd657af12010451045PyThreadState_Get (most recent call first):
Fatal Python error: in in : File PyThreadState_GetFatal Python error: Fatal Python error: _bootstrap_inner_bootstrap_innerthe function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)": Fatal Python error: PyThreadState_GetFatal Python error: PyThreadState_Get
Current thread 0x0000ffff818fc020 (most recent call first):
File "/home/vad/project/mmcv/mmcv/ops/focal_loss.py", line 59 in forward
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/autograd/function.py", line 575 in apply
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/losses/focal_loss.py", line 139 in sigmoid_focal_loss
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/losses/focal_loss.py", line 233 in forward
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD_head.py", line 1368 in loss_single
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/core/utils/misc.py", line 30 in multi_apply
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD_head.py", line 1691 in loss
File "/home/vad/project/mmcv/mmcv/runner/fp16_utils.py", line 208 in new_func
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 145 in forward_pts_train
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 246 in forward_train
File "/home/vad/project/mmcv/mmcv/runner/fp16_utils.py", line 208 in new_func
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 163 in forward
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/detectors/base.py", line 248 in train_step
File "/home/vad/project/mmcv/mmcv/parallel/distributed.py", line 63 in train_step
File "/home/vad/project/mmcv/mmcv/runner/epoch_based_runner.py", line 31 in run_iter
File "/home/vad/project/mmcv/mmcv/runner/epoch_based_runner.py", line 53 in train
File "/home/vad/project/mmcv/mmcv/runner/epoch_based_runner.py", line 136 in run
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/apis/mmdet_train.py", line 194 in custom_train_detector
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/apis/train.py", line 21 in custom_train_model
File "/home/vad/project/VAD/tools/train.py", line 264 in main
File "/home/vad/project/VAD/tools/train.py", line 290 in
您好，可能是分布式程序冲突，您可以清理一下环境中的残余进程后尝试
这个是啥原因呢？
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SigmoidFocalLoss.
由于算子异步下发，报错位置与实际出错的算子的位置未必一致，需要在环境中export ASCEND_LAUNCH_BLOCKING=1，将执行顺序改为同步，虽然会损失性能，但是能保证报错与实际出问题的地方一致，因此SigmoidFocalLoss的问题若出在使能ASCEND_LAUNCH_BLOCKING=1之前，则需添加同步后查看真正的报错位置
设置ASCEND_LAUNCH_BLOCKING=1，然后强制NPUS=1 训练报错信息
Fatal Python error: PyThreadState_Get: the function must be called with the GIL held, but the GIL is released (the current Python thread state is NULL)
Python runtime state: initialized
Thread 0x0000fff9e4f9f120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9e57af120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9e5fbf120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9e67cf120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9dffff120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9df7ef120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9defdf120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9de7cf120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9effff120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9f4cdf120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9f54ef120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fff9f5cff120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 327 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fffa056af120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 331 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/queue.py", line 180 in get
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/tensorboard/summary/writer/event_file_writer.py", line 269 in _run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/tensorboard/summary/writer/event_file_writer.py", line 244 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fffc1afdf120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 395 in _recv
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 430 in _recv_bytes
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 250 in recv
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/pool.py", line 579 in _handle_results
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fffc1b7ef120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/pool.py", line 531 in _handle_tasks
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fffc1bfff120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/selectors.py", line 415 in select
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 948 in wait
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/pool.py", line 502 in _wait_for_updates
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/pool.py", line 522 in _handle_workers
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 982 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000fffc99fe8120 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 395 in _recv
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 430 in _recv_bytes
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/connection.py", line 250 in recv
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/multiprocessing/managers.py", line 822 in _callmethod
File "<string>", line 2 in get
File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/multiprocess_util.py", line 91 in run
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/threading.py", line 1002 in _bootstrap
Current thread 0x0000ffffafcf7020 (most recent call first):
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/ops/focal_loss.py", line 59 in forward
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/autograd/function.py", line 575 in apply
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/losses/focal_loss.py", line 139 in sigmoid_focal_loss
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/losses/focal_loss.py", line 233 in forward
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD_head.py", line 1368 in loss_single
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/core/utils/misc.py", line 30 in multi_apply
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD_head.py", line 1691 in loss
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/runner/fp16_utils.py", line 208 in new_func
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 145 in forward_pts_train
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 246 in forward_train
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/runner/fp16_utils.py", line 208 in new_func
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 163 in forward
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747 in _call_impl
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736 in _wrapped_call_impl
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/detectors/base.py", line 248 in train_step
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/parallel/distributed.py", line 63 in train_step
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/runner/epoch_based_runner.py", line 31 in run_iter
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/runner/epoch_based_runner.py", line 53 in train
File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmcv/runner/epoch_based_runner.py", line 136 in run
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/apis/mmdet_train.py", line 194 in custom_train_detector
File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/apis/train.py", line 21 in custom_train_model
File "/home/vad/project/VAD/tools/train.py", line 264 in main
File "/home/vad/project/VAD/tools/train.py", line 290 in <module>
我们仓上提供的适配是python3.8和pytorch2.1.0的版本，目前看起来您使用的似乎是python3.11和pytorch2.5.1的版本，这应该会导致许多三方依赖的版本发生变化，可能会不适配导致异常，可以先尝试用python3.8和pytorch2.1.0重新搭建下环境。
在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。
环境信息
您是如何安装操作的
[rank6]: Traceback (most recent call last): [rank6]: File "/home/vad/project/VAD/tools/train.py", line 290, in <module> [rank6]: main() [rank6]: File "/home/vad/project/VAD/tools/train.py", line 264, in main [rank6]: custom_train_model( [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/apis/train.py", line 21, in custom_train_model [rank6]: custom_train_detector( [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/apis/mmdet_train.py", line 194, in custom_train_detector [rank6]: runner.run(data_loaders, cfg.workflow) [rank6]: File "/home/vad/project/mmcv/mmcv/runner/epoch_based_runner.py", line 136, in run [rank6]: epoch_runner(data_loaders[i], **kwargs) [rank6]: File "/home/vad/project/mmcv/mmcv/runner/epoch_based_runner.py", line 53, in train [rank6]: self.run_iter(data_batch, train_mode=True, **kwargs) [rank6]: File "/home/vad/project/mmcv/mmcv/runner/epoch_based_runner.py", line 31, in run_iter [rank6]: outputs = self.model.train_step(data_batch, self.optimizer, [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/mmcv/mmcv/parallel/distributed.py", line 63, in train_step [rank6]: output = self.module.train_step(*inputs[0], **kwargs[0]) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/models/detectors/base.py", line 248, in train_step [rank6]: losses = self(**data) [rank6]: ^^^^^^^^^^^^ [rank6]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl [rank6]: return self._call_impl(*args, **kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl [rank6]: return forward_call(*args, **kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 163, in forward [rank6]: return self.forward_train(**kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/mmcv/mmcv/runner/fp16_utils.py", line 208, in new_func [rank6]: return old_func(*args, **kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 246, in forward_train [rank6]: losses_pts = self.forward_pts_train(img_feats, gt_bboxes_3d, gt_labels_3d, [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD.py", line 145, in forward_pts_train [rank6]: losses = self.pts_bbox_head.loss(*loss_inputs, img_metas=img_metas) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/mmcv/mmcv/runner/fp16_utils.py", line 208, in new_func [rank6]: return old_func(*args, **kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD_head.py", line 1691, in loss [rank6]: losses_cls, losses_bbox, loss_traj, loss_traj_cls = multi_apply( [rank6]: ^^^^^^^^^^^^ [rank6]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/mmdet/core/utils/misc.py", line 30, in multi_apply [rank6]: return tuple(map(list, zip(*map_results))) [rank6]: ^^^^^^^^^^^^^^^^^ [rank6]: File "/home/vad/project/VAD/projects/mmdet3d_plugin/VAD/VAD_head.py", line 1372, in loss_single [rank6]: num_total_pos = loss_cls.new_tensor([num_total_pos]) [rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank6]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py", line 161, in decorated [rank6]: return fn(*args, **kwargs) [rank6]: ^^^^^^^^^^^^^^^^^^^ [rank6]: RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is SigmoidFocalLoss. [rank6]: Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, please set the environment variable ASCEND_LAUNCH_BLOCKING=1. [rank6]: Note: ASCEND_LAUNCH_BLOCKING=1 will force ops to run in synchronous mode, resulting in performance degradation. Please unset ASCEND_LAUNCH_BLOCKING in time after debugging. [rank6]: [ERROR] 2026-04-27-19:08:37 (PID:38388, Device:6, RankID:6) ERR00100 PTA call acl api failed. [rank6]: E19999: Inner Error! [rank6]: E19999: [PID: 38388] 2026-04-27-19:08:37.984.881 Assert ((append_atomic_index_ptr) != nullptr) failed[FUNC:TilingAppendAtomicIndex][FILE:tiling.cc][LINE:715] [rank6]: TraceBack (most recent call last): [rank6]: Assert ((TilingAppendAtomicIndex(tiling_data, atomic_index)) == ge::SUCCESS) failed[FUNC:TilingAppendDfxInfo][FILE:tiling.cc][LINE:742] [rank6]: [Exec][Op]Execute op failed. op type = SigmoidFocalLoss, ge result = 1343225857[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]欢迎加入社区，感谢您对社区的贡献 🎉!