已关闭

[Bug]:单卡训练场景采集host数据设置MSMEMSCOPE_HOSTMEM_PYSTACK_LIVE_BYTES=69552后会执行卡住 #144

王远兴创建于 12 天前关闭于 9 天前

王远兴成员

12 天前创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

API 令牌或密钥
密码或身份验证凭证
私有网址或接口地址
个人或机密数据
...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

例如：
- 操作系统
- 昇腾硬件信息
- CANN软件版本
- 安装的对应软件版本

🐛 问题描述

Thread 12 (Thread 0xfffd8a7c7040 (LWP 3568220) "acl_thread"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0xfffd8a7c4ba8, op=137, expected=0, futex_word=0x************ <_PyRuntime+428>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0xfffd8a7c4ba8, clockid=-1*********, expected=0, futex_word=0x************ <_PyRuntime+428>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word==0x <_PyRuntime+428>, expected=**************=0, clockid==1, abstime==0xfffd8a7c4ba8, private==0)--Type for more, q to quit, c to continue without paging--
at ./nptl/futex-internal.c:139
#3 0x0000ffffa20bcc10 in __pthread_cond_wait_common (abstime=0xfffd8a7c4ba8, clockid=1, mutex=0x*** <_PyRuntime+432>, cond=0x************ <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=0x************ <_PyRuntime+384>, mutex=0x************ <_PyRuntime+432>, abstime=0xfffd8a7c4ba8) at ./nptl/pthread_cond_wait.c:652
#5 0x0000************ in take_gil ()
#6 0x0000************ in PyEval_RestoreThread ()
#7 0x0000************ in PyGILState_Ensure ()
#8 0x0000ffffa1d7695c in Utility::PyInterpGuard::PyInterpGuard() () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libascend_leaks.so
#9 0x0000ffffa2300464 in hostmem::PyStackCapture::CaptureIfEnabled(hostmem::StackRecord&) () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libmsmemscope_host_mem_hook.so
#10 0x0000ffffa22fb890 in ?? () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libmsmemscope_host_mem_hook.so
#11 0x0000ffffa22fc724 in malloc () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libmsmemscope_host_mem_hook.so
#12 0x0000ffffa1edaeac in operator new(unsigned long) () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libascend_leaks.so
#13 0x0000ffffa1edaf0c in operator new(unsigned long, std::nothrow_t const&) () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libascend_leaks.so
#14 0x0000fffdb41c9f0c in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#15 0x0000fffdb41cbfa8 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#16 0x0000fffdb41cc264 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#17 0x0000fffdb44019d0 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#18 0x0000fffdb4402ec8 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#19 0x0000fffdb4025e08 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#20 0x0000fffdb40a61f8 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime_v100.so
#21 0x0000fffde621a1dc in rtsBinaryLoadFromData () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime.so
#22 0x0000fffde62d1914 in aclrtBinaryLoadFromDataImpl () from /usr/local/Ascend/cann-9.2.0/lib64/libruntime.so
#23 0x0000fffdd92cd6e8 in op::internal::OpKernelBin::BinLoadImpl(void*&) () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#24 0x0000fffdd92cdc10 in ?? () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#25 0x0000ffffa20c25d4 in __pthread_once_slow (once_control=0xaaab0f65100c, init_routine=0xffffa1f48870 <__once_proxy>) at ./nptl/pthread_once.c:116
#26 0x0000fffdd92c771c in op::internal::OpKernelBin::BinLoad() () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#27 0x0000fffdd92be48c in op::internal::OpKernelBin::Launch(void*, op::OpArgContext*) () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#28 0x0000fffdd92c3d2c in op::AiCoreKernelLauncher::Launch() () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#29 0x0000fffdd92b4654 in aclOpExecutor::Run() () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#30 0x0000fffdd92b61dc in CommonOpExecutorRun(void*, unsigned long, aclOpExecutor*, void*) () from /usr/local/Ascend/cann-9.2.0/lib64/libnnopbase.so
#31 0x0000fffdbb287a20 in aclnnForeachAddListV2 () from /usr/local/Ascend/cann-9.2.0/lib64/libopapi_nn.so
#32 0x0000fffded2c22a0 in ?? () from /root/miniconda3/envs/py_39/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so
#33 0x0000fffdef5d4adc in ?? () from /root/miniconda3/envs/py_39/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so
#34 0x0000fffdec61c0b4 in ?? () from /root/miniconda3/envs/py_39/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so
#35 0x0000fffdec620a7c in ?? () from /root/miniconda3/envs/py_39/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so
#36 0x0000fffdec61ad1c in ?? () from /root/miniconda3/envs/py_39/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so
#37 0x0000ffffa1f534bc in ?? () from /usr/local/Ascend/cann-9.2.0/tools/msmemscope/lib64/libascend_leaks.so
#38 0x0000ffffa20bd5c8 in start_thread (arg=0x0) at ./nptl/pthread_create.c:442
#39 0x0000ffffa2125edc in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:79

欢迎加入社区，感谢您对社区的贡献 🎉!

王远兴成员

12 天前关联了里程碑：MindStudio 26.2.0

王远兴成员

12 天前添加了label：bug, medium-priority

tanxiangyuu成员

12 天前评论：

👋 您好,欢迎向MindStudio MemScope提交 Issue！
我们已收到您的反馈，感谢你对开源社区的支持。🎉

📅处理时效：维护团队将在24小时内查看并回复您的问题（工作日）。
🔍自助查询：在等待期间，建议您先查阅以下资料，可能已有解决方案：

📖 MindStudio MemScope官方文档

请确保 Issue 描述清晰，包含复现步骤和日志，这将帮助我们更快定位问题。谢谢！

M Martin_M成员

10 天前添加了label：resolved

王远兴成员

9 天前评论：

(py_39) [root@90 c]# make
/home/w/B066/Ascend/cann-9.2.0/bin/ccec -xcce -O2 -std=c++17 --cce-enable-sanitizer -g --cce-aicore-arch=dav-c220-vec -I/home/w/B066/Ascend/cann-9.2.0/compiler/tikcpp/tikcfw -I/home/w/B066/Ascend/cann-9.2.0/compiler/tikcpp/tikcfw/impl -I/home/w/B066/Ascend/cann-9.2.0/compiler/tikcpp/tikcfw/interface -I/home/w/B066/Ascend/cann-9.2.0/include -o test_kernel.o -c test_kernel.cpp
g++ -O2 -std=c++17 -I/home/w/B066/Ascend/cann-9.2.0/include -o main.o -c main.cpp
/home/w/B066/Ascend/cann-9.2.0/bin/ccec --cce-fatobj-link --cce-enable-sanitizer --cce-aicore-arch=dav-c220-vec -o test_kernel.fatbin test_kernel.o main.o -L/home/w/B066/Ascend/cann-9.2.0/lib64 -lruntime -lascendcl -lms_tools_ext -lstdc++
(py_39) [root@90 c]# msmemscope ./
main.cpp main.o Makefile test_kernel.cpp test_kernel.fatbin test_kernel.o
(py_39) [root@90 c]# msmemscope ./test_kernel.fatbin

               >>>>>   MindStudio   <<<<<
THE END-TO-END TOOLCHAIN TO UNLEASH HUAWEI ASCEND COMPUTE

=================================================================

[msmemscope] Info: Config into file /home/w********/B066/msmemscope/example/leaks/c/memscopeDumpResults/msmemscope_796637_202048_ascend/config.json
[msmemscope] Info: logging into file /home/w/B066/msmemscope/example/leaks/c/memscopeDumpResults/msmemscope_796637_202048_ascend/msmemscope_logs/msmemscope_202048.log
[msmemscope] Info: logging into file /home/w********/B066/msmemscope/example/leaks/c/memscopeDumpResults/msmemscope_796637_202048_ascend/msmemscope_logs/msmemscope_202048.log
[msmemscope] Info: create file /home/w********/B066/msmemscope/example/leaks/c/memscopeDumpResults/msmemscope_796637_202048_ascend/device_0/dump/memscope_dump_202058.csv.
[msmemscope] Info: Fflush temporary cache events to file!

王远兴成员

9 天前评论：

回归验证通过

王远兴成员

9 天前 issue状态由 TODO 改变为 DONE

王远兴成员

9 天前关闭了 issue