Star892
1410
代码
代码
Issues1691
Pull Requests1755
Actions
项目讨论
Wiki
分析
Star892
1410
已关闭
[Bug-Report|缺陷反馈]: freeze on HcclAllreduce during graph capturing on 2 nodes #1415
vserov创建于  2025年11月4日关闭于  7月4日
V
vserov
2025年11月4日 创建

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

Original issue found on SGLang: https://github.com/sgl-project/sglang/issues/12187
Issue has been found on GLM45 model, but applicable for Qwen3 MoE models (Qwen3-30B-A3B).
On non-MoE models issue is not reproduced.

Environment / 环境信息 (Mandatory / 必填)

Docker images:lmsysorg/sglang:dsv32-a2
910b A2 64g*8 *2
hdk try:24.1.rc1 /25.0.rc1.1

absl-py 2.3.1
aiohappyeyeballs 2.6.1
aiohttp 3.12.15
aiosignal 1.4.0
airportsdata 20250909
annotated-types 0.7.0
anthropic 0.71.0
anyio 4.11.0
astor 0.8.1
asttokens 3.0.0
attrs 24.2.0
auto_tune 0.1.0
blake3 1.0.6
blobfile 3.0.0
build 1.3.0
cachetools 6.2.0
certifi 2025.7.14
cffi 1.17.1
charset-normalizer 3.4.2
click 8.3.0
cloudpickle 3.1.1
compressed-tensors 0.9.3
custom-ops 1.0
Cython 3.1.2
dataflow 0.0.1
datasets 4.1.1
decorator 5.1.1
decord2 2.0.0
deep-ep 1.0.0+41ba5d74
Deprecated 1.2.18
depyf 0.18.0
dill 0.4.0
diskcache 5.6.3
distro 1.9.0
dnspython 2.8.0
docstring_parser 0.17.0
einops 0.8.1
email-validator 2.3.0
execnet 2.1.1
executing 2.2.1
fastapi 0.118.0
fastapi-cli 0.0.13
fastapi-cloud-cli 0.2.1
filelock 3.19.1
frozenlist 1.7.0
fsspec 2025.9.0
gguf 0.17.1
googleapis-common-protos 1.70.0
grpcio 1.75.1
grpcio-reflection 1.75.1
grpcio-tools 1.75.1
h11 0.16.0
hccl 0.1.0
hccl_parser 0.1
hf_transfer 0.1.9
hf-xet 1.1.10
httpcore 1.0.9
httptools 0.6.4
httpx 0.28.1
huggingface-hub 0.35.3
idna 3.10
importlib_metadata 8.0.0
iniconfig 2.1.0
interegular 0.3.3
ipython 9.6.0
ipython_pygments_lexers 1.1.1
jedi 0.19.2
Jinja2 3.1.6
jiter 0.11.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
lark 1.2.2
llguidance 0.7.30
llm_datadist 0.0.1
lm-format-enforcer 0.10.12
lxml 6.0.2
markdown-it-py 4.0.0
MarkupSafe 3.0.3
matplotlib-inline 0.1.7
mdurl 0.1.2
mf-adapter 1.0.0
mistral_common 1.8.5
modelscope 1.30.0
mpmath 1.3.0
msgspec 0.19.0
msobjdump 0.1.0
multidict 6.6.4
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.5
ninja 1.13.0
numpy 1.26.4
nvidia-ml-py 13.580.82
op_compile_tool 0.1.0
op_gen 0.1
op_test_frame 0.1
opc_tool 0.1.0
openai 1.99.1
openai-harmony 0.0.4
opencv-python-headless 4.12.0.88
opentelemetry-api 1.26.0
opentelemetry-exporter-otlp 1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc 1.26.0
opentelemetry-exporter-otlp-proto-http 1.26.0
opentelemetry-proto 1.26.0
opentelemetry-sdk 1.26.0
opentelemetry-semantic-conventions 0.47b0
opentelemetry-semantic-conventions-ai 0.4.13
orjson 3.11.3
outlines 0.1.11
outlines_core 0.1.26
packaging 25.0
pandas 2.3.2
parso 0.8.5
partial-json-parser 0.2.1.1.post6
pathlib2 2.3.7.post1
pexpect 4.9.0
pillow 11.3.0
pip 25.2
pluggy 1.6.0
prometheus_client 0.23.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.52
propcache 0.3.2
protobuf 6.33.0
psutil 6.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.1
pyarrow 21.0.0
pybase64 1.4.2
pybind11 3.0.1
pycountry 24.6.1
pycparser 2.22
pycryptodomex 3.23.0
pydantic 2.11.9
pydantic_core 2.33.2
pydantic-extra-types 2.10.5
Pygments 2.19.2
pynvml 13.0.1
pyproject_hooks 1.2.0
pytest 8.3.2
pytest-xdist 3.6.1
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-json-logger 3.3.0
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.2
pyzmq 27.1.0
referencing 0.36.2
regex 2025.9.18
requests 2.32.4
rich 14.1.0
rich-toolkit 0.15.1
rignore 0.6.4
rpds-py 0.27.1
safetensors 0.6.2
schedule_search 0.0.1
scipy 1.13.1
semantic-version 2.10.0
sentencepiece 0.2.1
sentry-sdk 2.39.0
setproctitle 1.3.7
setuptools 65.5.0
setuptools-rust 1.12.0
sgl-kernel-npu 0.1.0
sglang 0.5.4 /workspace/sglang/python
sglang-router 0.1.9
shellingham 1.5.4
show_kernel_debug_data 0.1.0
six 1.17.0
sniffio 1.3.1
soundfile 0.13.1
stack-data 0.6.3
starlette 0.48.0
sympy 1.13.1
te 0.4.0
tiktoken 0.11.0
timm 1.0.16
tokenizers 0.22.1
torch 2.6.0+cpu
torch-npu 2.6.0.post3
torchao 0.9.0
torchvision 0.21.0
tqdm 4.67.1
traitlets 5.14.3
transformers 4.57.1
triton-ascend 3.2.0+gitb0ea0850
typer 0.19.2
typing_extensions 4.15.0
typing-inspection 0.4.1
tzdata 2025.2
urllib3 2.5.0
uvicorn 0.37.0
uvloop 0.21.0
vllm 0.8.5.post1+empty /workspace/sglang/vllm
watchfiles 1.1.0
wcwidth 0.2.14
websockets 15.0.1
wheel 0.45.1
wrapt 1.17.3
xgrammar 0.1.25
xxhash 3.5.0
yarl 1.20.1
zipp 3.23.0

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

Execute on 2 nodes
master:

export HCCL_IF_IP=xxx
export GLOO_SOCKET_IFNAME="bond0" # network card name
export TP_SOCKET_IFNAME="bond0"
export HCCL_SOCKET_IFNAME="bond0"
export OMP_PROC_BIND=false
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE=AIV

python -m sglang.launch_server --model-path /glm-4.5/w8a8/ --trust-remote-code --nnodes 2 --node-rank 0 --attention-backend ascend --host=0.0.0.0 --port=8080 --max-running-requests 32 --context-length 8192 --disable-radix-cache --chunked-prefill-size 32768 --max-prefill-tokens 28000 --tp-size 16 --cuda-graph-max-bs 8 --mem-fraction-static 0.9 --reasoning-parser=qwen3 --tool-call-parser=glm45 --dist-init-addr=10.2.0.87:20000 --quantization w8a8_int8

sub:
export HCCL_IF_IP=xxxx
export GLOO_SOCKET_IFNAME="bond0" # network card name
export TP_SOCKET_IFNAME="bond0"
export HCCL_SOCKET_IFNAME="bond0"
export OMP_PROC_BIND=false
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE=AIV

python -m sglang.launch_server --model-path /glm-4.5/w8a8/ --trust-remote-code --nnodes 2 --node-rank 1 --attention-backend ascend --host=0.0.0.0 --port=8080 --max-running-requests 32 --context-length 8192 --disable-radix-cache --chunked-prefill-size 32768 --max-prefill-tokens 28000 --tp-size 16 --cuda-graph-max-bs 8 --mem-fraction-static 0.9 --dist-init-addr=10.2.0.87:20000 --quantization w8a8_int8

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Should not freeze on HcclAllreduce during graph capture

Log from first node:
[INFO] APP(190317,):2025-11-04-00:41:22.833.847 [log_inner.cpp:82]193231 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:RecordEventFunc:510: "[PTA]:"Event: aclrtRecordEvent dequeue is successfully executed, stream=0xaaab0ab42770, event=0xaaab1199df20""
[DEBUG] APP(190319,):2025-11-04-00:41:22.833.885 [log_inner.cpp:85]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeFinalizeRoutingV2 Run.""
[DEBUG] APP(190319,):2025-11-04-00:41:22.833.888 [log_inner.cpp:85]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeFinalizeRoutingV2 with custom handle""
[INFO] APP(190320,):2025-11-04-00:41:22.833.902 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:createEvent:185: "[PTA]:"Event: aclrtCreateEventWithFlag is successfully executed, event=0xaaab5078afc0""
[INFO] APP(190320,):2025-11-04-00:41:22.833.910 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MaybeSetDevice:141: "[PTA]:"MaybeSetDevice: NPU device 5 has not been initialized! We will set targetDeviceIndex.""
[INFO] APP(190320,):2025-11-04-00:41:22.833.921 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MaybeSetDevice:141: "[PTA]:"MaybeSetDevice: NPU device 5 has not been initialized! We will set targetDeviceIndex.""
[INFO] APP(190320,):2025-11-04-00:41:22.833.925 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:LaunchWaitTask:161: "[PTA]:"Event: LaunchWaitTask is successfully executed, event=0xaaab5078afc0""
[INFO] APP(190320,):2025-11-04-00:41:22.833.929 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MaybeSetDevice:141: "[PTA]:"MaybeSetDevice: NPU device 5 has not been initialized! We will set targetDeviceIndex.""
[INFO] APP(190320,):2025-11-04-00:41:22.833.932 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:THNPEvent_wait:91: "[PTA]:"Event: wait api is successfully executed, event=0xaaab5078afc0""
[INFO] APP(190320,):2025-11-04-00:41:22.833.940 [log_inner.cpp:82]190320 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MakeSureQueueEmpty:249: "[PTA]:"Begin to makesure taskqueue empty.""
[DEBUG] APP(190320,):2025-11-04-00:41:22.833.961 [log_inner.cpp:85]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeFinalizeRoutingV2 Run.""
[DEBUG] APP(190320,):2025-11-04-00:41:22.833.965 [log_inner.cpp:85]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeFinalizeRoutingV2 with custom handle""
[INFO] APP(190317,):2025-11-04-00:41:22.833.971 [log_inner.cpp:82]193231 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:WaitEventFunc:534: "[PTA]:"Event: aclrtStreamWaitEvent dequeue is successfully executed, stream=0xaaab0adf1bb0, event=0xaaab1199df20""
[DEBUG] APP(190317,):2025-11-04-00:41:22.833.976 [log_inner.cpp:85]193231 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op HcclAllreduce Run.""
[DEBUG] APP(190317,):2025-11-04-00:41:22.833.979 [log_inner.cpp:85]193231 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op HcclAllreduce with custom handle""
[INFO] APP(190319,):2025-11-04-00:41:22.834.079 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:createEvent:185: "[PTA]:"Event: aclrtCreateEventWithFlag is successfully executed, event=0xaaab5a884750""
[INFO] APP(190319,):2025-11-04-00:41:22.834.088 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MaybeSetDevice:141: "[PTA]:"MaybeSetDevice: NPU device 4 has not been initialized! We will set targetDeviceIndex.""
[INFO] APP(190319,):2025-11-04-00:41:22.834.099 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MaybeSetDevice:141: "[PTA]:"MaybeSetDevice: NPU device 4 has not been initialized! We will set targetDeviceIndex.""
[INFO] APP(190319,):2025-11-04-00:41:22.834.103 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:LaunchWaitTask:161: "[PTA]:"Event: LaunchWaitTask is successfully executed, event=0xaaab5a884750""
[INFO] APP(190319,):2025-11-04-00:41:22.834.108 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MaybeSetDevice:141: "[PTA]:"MaybeSetDevice: NPU device 4 has not been initialized! We will set targetDeviceIndex.""
[INFO] APP(190319,):2025-11-04-00:41:22.834.111 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:THNPEvent_wait:91: "[PTA]:"Event: wait api is successfully executed, event=0xaaab5a884750""
[INFO] APP(190319,):2025-11-04-00:41:22.834.119 [log_inner.cpp:82]190319 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MakeSureQueueEmpty:249: "[PTA]:"Begin to makesure taskqueue empty.""
[DEBUG] APP(190319,):2025-11-04-00:41:22.834.251 [log_inner.cpp:85]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:DecreaseUnrecordedCount:126: "[PTA]:"Event: unrecorded count decrease, now=0.""
[INFO] APP(190319,):2025-11-04-00:41:22.834.256 [log_inner.cpp:82]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:RecordEventFunc:510: "[PTA]:"Event: aclrtRecordEvent dequeue is successfully executed, stream=0xaaab47aee9f0, event=0xaaab4d6c4200""
[DEBUG] APP(190320,):2025-11-04-00:41:22.834.311 [log_inner.cpp:85]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:DecreaseUnrecordedCount:126: "[PTA]:"Event: unrecorded count decrease, now=0.""
[INFO] APP(190320,):2025-11-04-00:41:22.834.315 [log_inner.cpp:82]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:RecordEventFunc:510: "[PTA]:"Event: aclrtRecordEvent dequeue is successfully executed, stream=0xaaab3c8e8d60, event=0xaaab436bec60""
[INFO] APP(190319,):2025-11-04-00:41:22.834.377 [log_inner.cpp:82]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:WaitEventFunc:534: "[PTA]:"Event: aclrtStreamWaitEvent dequeue is successfully executed, stream=0xaaab46a8b220, event=0xaaab4d6c4200""
[DEBUG] APP(190319,):2025-11-04-00:41:22.834.382 [log_inner.cpp:85]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op HcclAllreduce Run.""
[DEBUG] APP(190319,):2025-11-04-00:41:22.834.385 [log_inner.cpp:85]193236 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op HcclAllreduce with custom handle""
[INFO] APP(190320,):2025-11-04-00:41:22.834.431 [log_inner.cpp:82]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:WaitEventFunc:534: "[PTA]:"Event: aclrtStreamWaitEvent dequeue is successfully executed, stream=0xaaab3cb98250, event=0xaaab436bec60""
[DEBUG] APP(190320,):2025-11-04-00:41:22.834.436 [log_inner.cpp:85]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op HcclAllreduce Run.""
[DEBUG] APP(190320,):2025-11-04-00:41:22.834.439 [log_inner.cpp:85]193280 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op HcclAllreduce with custom handle""

Log from second node:
[INFO] APP(155885,):2025-11-04-00:38:23.232.522 [log_inner.cpp:82]155885 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:THNPEvent_wait:91: "[PTA]:"Event: wait api is successfully executed, event=0xaaab27270390""
[DEBUG] APP(155890,):2025-11-04-00:38:23.232.526 [log_inner.cpp:85]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeFinalizeRoutingV2 Run.""
[DEBUG] APP(155890,):2025-11-04-00:38:23.232.532 [log_inner.cpp:85]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeFinalizeRoutingV2 with custom handle""
[INFO] APP(155885,):2025-11-04-00:38:23.232.533 [log_inner.cpp:82]155885 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:MakeSureQueueEmpty:249: "[PTA]:"Begin to makesure taskqueue empty.""
[DEBUG] APP(155891,):2025-11-04-00:38:23.232.661 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnSwiGlu Run.""
[DEBUG] APP(155891,):2025-11-04-00:38:23.232.669 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnSwiGlu with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.232.688 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnArange Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.232.695 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnArange with custom handle""
[DEBUG] APP(155891,):2025-11-04-00:38:23.232.999 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnGroupedMatmulV5 Run.""
[DEBUG] APP(155891,):2025-11-04-00:38:23.233.004 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnGroupedMatmulV5 with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.233.068 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnInplaceCopy Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.233.073 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnInplaceCopy with custom handle""
[DEBUG] APP(155890,):2025-11-04-00:38:23.233.155 [log_inner.cpp:85]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:DecreaseUnrecordedCount:126: "[PTA]:"Event: unrecorded count decrease, now=0.""
[INFO] APP(155890,):2025-11-04-00:38:23.233.159 [log_inner.cpp:82]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:RecordEventFunc:510: "[PTA]:"Event: aclrtRecordEvent dequeue is successfully executed, stream=0xaaab36793c00, event=0xaaab3d56af90""
[INFO] APP(155890,):2025-11-04-00:38:23.233.357 [log_inner.cpp:82]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:WaitEventFunc:534: "[PTA]:"Event: aclrtStreamWaitEvent dequeue is successfully executed, stream=0xaaab36a42f30, event=0xaaab3d56af90""
[DEBUG] APP(155890,):2025-11-04-00:38:23.233.363 [log_inner.cpp:85]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op HcclAllreduce Run.""
[DEBUG] APP(155890,):2025-11-04-00:38:23.233.369 [log_inner.cpp:85]156921 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op HcclAllreduce with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.233.456 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeInitRouting Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.233.460 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeInitRouting with custom handle""
[DEBUG] APP(155891,):2025-11-04-00:38:23.233.490 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeFinalizeRoutingV2 Run.""
[DEBUG] APP(155891,):2025-11-04-00:38:23.233.495 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeFinalizeRoutingV2 with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.233.939 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeComputeExpertTokens Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.233.944 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeComputeExpertTokens with custom handle""
[DEBUG] APP(155891,):2025-11-04-00:38:23.234.168 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:DecreaseUnrecordedCount:126: "[PTA]:"Event: unrecorded count decrease, now=0.""
[INFO] APP(155891,):2025-11-04-00:38:23.234.173 [log_inner.cpp:82]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:RecordEventFunc:510: "[PTA]:"Event: aclrtRecordEvent dequeue is successfully executed, stream=0xaaab215ea600, event=0xaaab27ebdce0""
[DEBUG] APP(155885,):2025-11-04-00:38:23.234.338 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnCast Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.234.344 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnCast with custom handle""
[INFO] APP(155891,):2025-11-04-00:38:23.234.377 [log_inner.cpp:82]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:WaitEventFunc:534: "[PTA]:"Event: aclrtStreamWaitEvent dequeue is successfully executed, stream=0xaaab21699b20, event=0xaaab27ebdce0""
[DEBUG] APP(155891,):2025-11-04-00:38:23.234.384 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op HcclAllreduce Run.""
[DEBUG] APP(155891,):2025-11-04-00:38:23.234.387 [log_inner.cpp:85]157211 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op HcclAllreduce with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.234.654 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnGroupedMatmulV5 Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.234.660 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnGroupedMatmulV5 with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.235.141 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnSwiGlu Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.235.144 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnSwiGlu with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.235.415 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnGroupedMatmulV5 Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.235.421 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnGroupedMatmulV5 with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.235.850 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op aclnnMoeFinalizeRoutingV2 Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.235.857 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op aclnnMoeFinalizeRoutingV2 with custom handle""
[DEBUG] APP(155885,):2025-11-04-00:38:23.236.365 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:DecreaseUnrecordedCount:126: "[PTA]:"Event: unrecorded count decrease, now=0.""
[INFO] APP(155885,):2025-11-04-00:38:23.236.373 [log_inner.cpp:82]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:RecordEventFunc:510: "[PTA]:"Event: aclrtRecordEvent dequeue is successfully executed, stream=0xaaab13612610, event=0xaaab1a0ef6d0""
[INFO] APP(155885,):2025-11-04-00:38:23.236.527 [log_inner.cpp:82]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:WaitEventFunc:534: "[PTA]:"Event: aclrtStreamWaitEvent dequeue is successfully executed, stream=0xaaab132c0340, event=0xaaab1a0ef6d0""
[DEBUG] APP(155885,):2025-11-04-00:38:23.236.536 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:424: "[PTA]:"Op HcclAllreduce Run.""
[DEBUG] APP(155885,):2025-11-04-00:38:23.236.540 [log_inner.cpp:85]156948 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:ExecFuncOpApi:428: "[PTA]:"Exec Op HcclAllreduce with custom handle""

Special notes for this issue/备注 (Optional / 选填)

likedislike
H
huangyunlong成员
2025年11月4日 评论：

如果是卡住了，麻烦提供下对应的堆栈信息

likedislike
V
vserov
2025年11月6日 评论：
H
huangyunlong成员
2025年11月6日 评论：

I see it is currently executing, please collect the stack when it gets stuck.
If hccl allreduce gets stuck, won't it report an error in the end. If there is an error, please provide detailed error logs

likedislike
V
vserov
2025年11月6日 评论：

Stack trace "stack after allreduce call" is collected after hccl allreduce gets stuck. There is no error, and other code can wait execution of allreduce more then hour.

likedislike
H
huangyunlong成员
2025年11月6日 评论：

You can collect the stack at regular intervals to see if there are any changes.
Reduce HCCL_EXEC_TIMEOUT and check if hccl reports a timeout error

likedislike
V
vserov
2025年11月10日 评论：

set HCCL_EXEC_TIMEOUT to 5, wait more then 10 minutes, no timeout error hasn't been observed. (export HCCL_EXEC_TIMEOUT=5)
stacks from all processes have been collected.
stack from rang 0 ( node 1 )
stack from rang 1 ( node 1 )
stack from rang 2 ( node 2 )
stack from rang 3 ( node 2 )

likedislike
H
huangyunlong成员
2025年11月11日 评论：

1、Is there only 4 processes？
2、rang3 CannKbInit hangs. It may have just initialized，you cann see the python stack with py-spy dump -p {pid}
3、If the information provided above is complete and accurate，rang 0, 1, 2 on hcclallreduce，rang 3 hangs on init, rang 0, 1, 2 may timeout until HCCL_EXEC_TIMEOUT

likedislike
V
vserov
2025年11月19日 评论：
chenrayray
chenrayray成员
7月4日 评论：

您好，本Issue因长期无有效互动，现暂时关闭。如问题依然存在，请重新打开并提供最新复现信息，我们会继续跟进。感谢理解！

likedislike
chenrayraychenrayray成员
7月4日 issue状态由 TODO 改变为 DONE
chenrayraychenrayray成员
7月4日 关闭了 issue
ascend-robotascend-robot成员
7月4日 添加了label：resolved