2026-01-10 16:30:26,776 - msmodelslim - INFO - Calibration start!
0%| | 0/44 [00:00<?, ?it/s][W compiler_depend.ts:116] Warning: Driver Version: ������ is invalid or not supported yet. (function operator())
0%| | 0/44 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 533, in run
self._run(int_infer=int_infer)
File "/usr/local/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 977, in _run
self.run_calib_mode()
File "/usr/local/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 588, in run_calib_mode
self.model(*data)
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 821, in forward
outputs: BaseModelOutputWithPast = self.model(
^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/utils/generic.py", line 965, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/transformers/models/llama/modeling_llama.py", line 548, in forward
position_embeddings = self.rotary_emb(hidden_states, position_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/hooks.py", line 161, in new_forward
args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/accelerate_adapter/hook_adapter.py", line 234, in pre_forward
args, kwargs = self.old_hook.pre_forward(module, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/hooks.py", line 356, in pre_forward
return send_to_device(args, self.execution_device), send_to_device(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/utils/operations.py", line 158, in send_to_device
return honor_type(
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/utils/operations.py", line 90, in honor_type
return type(obj)(generator)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/utils/operations.py", line 159, in <genexpr>
tensor, (send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys) for t in tensor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/accelerate/utils/operations.py", line 186, in send_to_device
return tensor.to(device, non_blocking=non_blocking)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py", line 165, in decorated
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch_npu/utils/_to_int.py", line 34, in decorated
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
RuntimeError: copy_d2d:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:285 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507013
[ERROR] 2026-01-10-16:30:26 (PID:123445, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
The error from device(0), serial number is 2. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000000880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00001003d000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:780]
rtStreamSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999[PID: 123445] 2026-01-10-16:30:26.941.004 (EH9999): synchronize stream with timeout failed, runtime result = 507013[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
TraceBack (most recent call last):
DEVICE[0] PID[123445]:
EXCEPTION STREAM:
Exception info:TGID=515234, model id=65535, stream id=3, stream phase=SCHEDULE
Message info[0]:RTS_HWTS: hwts sdma error, slot_id=14, stream_id=3
Other info[0]:time=2026-01-10-16:30:26.689.916, function=int_process_hwts_sdma_error, line=2070, error code=0x20b
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home1/gitrepo/msmodelslim/example/Llama/quant_llama.py", line 400, in <module>
quantifier.convert(tokenized_calib_data, save_directory, args.disable_level, part_file_size=args.part_file_size, \
File "/home1/gitrepo/msmodelslim/example/Llama/quant_llama.py", line 293, in convert
calibrator.run()
File "/usr/local/lib64/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/msmodelslim/pytorch/llm_ptq/llm_ptq_tools/quant_tools.py", line 535, in run
raise Exception("Please check the model and configuration.", ex) from ex
Exception: ('Please check the model and configuration.', RuntimeError("copy_d2d:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:285 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(copy_stream), error code is 507013\n[ERROR] 2026-01-10-16:30:26 (PID:123445, Device:0, RankID:-1) ERR00100 PTA call acl api failed\n[Error]: System Direct Memory Access (DMA) hardware execution error. \n Rectify the fault based on the error information in the ascend log.\nEH9999: Inner Error!\n The error from device(0), serial number is 2. there is a sdma error, sdma channel is 0, the channel exist the following problems: The SMMU returns a Terminate error during page table translation.. the value of CQE status is 2. the description of CQE status: When the SQE translates a page table, the SMMU returns a Terminate error.it's config include: setting1=0xc000000880e0000, setting2=0xff009000ff004c, setting3=0, sq base addr=0x800d00001003d000[FUNC:ProcessSdmaErrorInfo][FILE:device_error_proc.cc][LINE:780]\n rtStreamSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]\nEH9999[PID: 123445] 2026-01-10-16:30:26.941.004 (EH9999): synchronize stream with timeout failed, runtime result = 507013[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]\n TraceBack (most recent call last):\n\n\nDEVICE[0] PID[123445]: \nEXCEPTION STREAM:\n Exception info:TGID=515234, model id=65535, stream id=3, stream phase=SCHEDULE\n Message info[0]:RTS_HWTS: hwts sdma error, slot_id=14, stream_id=3\n Other info[0]:time=2026-01-10-16:30:26.689.916, function=int_process_hwts_sdma_error, line=2070, error code=0x20b"))
[W compiler_depend.ts:545] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EI9999: Inner Error!
EI9999[PID: 123445] 2026-01-10-16:30:26.963.163 (EI9999): Memory async copy failed, device_id=0, stream_id=3, task_id=7, flip_num=0, copy_type=2, memcpy_type=0, copy_data_type=0, length=352256[FUNC:GetError][FILE:stream.cc][LINE:1191]
TraceBack (most recent call last):
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
(function npuSynchronizeUsedDevices)
[W compiler_depend.ts:527] Warning: NPU warning, error code is 507013[Error]:
[Error]: System Direct Memory Access (DMA) hardware execution error.
Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
rtDeviceSynchronizeWithTimeout execute failed, reason=[sdma copy error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999[PID: 123445] 2026-01-10-16:30:26.965.824 (EH9999): wait for compute device to finish failed, runtime result = 507013.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
TraceBack (most recent call last):
(function npuSynchronizeDevice)
在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。
环境信息
硬件：300I Duo
操作系统 openEuler 24.03
CANN: 8.3.RC2
driver: 24.1.rc2
MindIE-ATB: master 30920dbaf71a550f04589d83ee96e6ef3735317b
ATB: 8.3.RC2
msmodelslim: 8.2.0
使用模型：LLama3.1-8B-Instruct
使用场景及问题
运行后报错：
进行w8a8的量化则不会出现问题，另外，我在qwen2.5-7b上也碰到过类似的问题，w8a8正常，w4a16就会出现类似的错误，请问这要如何解决？
欢迎加入社区，感谢您对社区的贡献 🎉!