已关闭
[Bug-Report|缺陷反馈]: 单边通信并发调用HcommChannelCreate导致coredump #670
lizhuolong创建于 7月29日关闭于 9月8日
7月29日 修改了issue 的描述
7月29日 关联了pull request:[bugfix]Concurrent resource access under the same endpoint
7月29日 将 JC_TJC 设为负责人
7月29日 关联了pull request:[bugfix]Concurrent resource access under the same endpoint
7月30日 关联了pull request:[bugfix]for channel get all memhandle
7月30日 关联了pull request:[bugfix]for channel get all memhandle


Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.
Describe the current behavior / 问题描述 (Mandatory / 必填)
PD分离,模型场景,单边通信hixl,多线程调用HcommChannelCreate接口,在同一个endpoint下并发建链,原设计未对一个endpoint下资源保护,导致的竟态访问,使得memhandle指向内容为空,访问引发的coredump问题。
Environment / 环境信息 (Mandatory / 必填)
昇腾950
Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
16P/32P的PD分离GLM5.1模型场景
Describe the expected behavior / 预期结果 (Mandatory / 必填)
单边通信hixl,多线程调用HcommChannelCreate接口,代码行为上支持同一个endpoint下并发建链。
Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
(gdb) bt full
#0 0x0000ffff7084c404 in hcomm::HostCpuUrmaChannel::ParseInputParam() ()
from /usr/local/Ascend/ascend-toolkit/latest/lib64/libhcomm.so
No symbol table info available.
#1 0x0000ffff7084db7c in hcomm::HostCpuUrmaChannel::Init() ()
from /usr/local/Ascend/ascend-toolkit/latest/lib64/libhcomm.so
No symbol table info available.
#2 0x0000ffff707e4308 in hcomm::Channel::CreateChannel(void*, CommEngine, HcommChannelDesc, std::shared_ptrhcomm::Channel&) () from /usr/local/Ascend/ascend-toolkit/latest/lib64/libhcomm.so
No symbol table info available.
#3 0x0000ffff707ec198 in hcomm::ChannelProcess::CreateChannelsLoop(void*, CommEngine, HcommChannelDesc*, unsigned int, unsigned long*) () from /usr/local/Ascend/ascend-toolkit/latest/lib64/libhcomm.so
No symbol table info available.
#4 0x0000ffff706f3ffc in HcommChannelCreate () from /usr/local/Ascend/ascend-toolkit/latest/lib64/libhcomm.so
No symbol table info available.
#5 0x0000fffd84227700 in hixl::Channel::Create(void*, HcommChannelDesc&, CommEngine, unsigned int) ()
from /usr/local/Ascend/ascend-toolkit/latest/lib64/libcann_hixl.so
No symbol table info available.
#6 0x0000fffd8422cb14 in hixl::Endpoint::CreateChannel(hixl::ChannelDesc const&, unsigned long&, unsigned int) ()
from /usr/local/Ascend/ascend-toolkit/latest/lib64/libcann_hixl.so
No symbol table info available.
#7 0x0000fffd84255488 in hixl::HixlCSServer::CreateChannel(int, char const*, unsigned long) ()
from /usr/local/Ascend/ascend-toolkit/latest/lib64/libcann_hixl.so
No symbol table info available.
Special notes for this issue/备注 (Optional / 选填)
(gdb) info registers
x0 0x0 0
x1 0x0 0
x2 0xc980322b7bcf7500 -3927083712724896512
x3 0x0 0
x4 0x4 4
x5 0xfffc9c354250 281460417577552
x6 0xa23d51f96644c109 -6756153734381387511
x7 0x1d 29
x8 0x101010101010101 72340172838076673
x9 0x0 0
x10 0x35 53
x11 0xfffcafffe1b0 281460749623728
x12 0xffffb6704ab4 281473742555828
x13 0x0 0
x14 0x1 1
x15 0x10004000c 4295229452
x16 0xffff70bba0c8 281472573087944
x17 0xffff717e6c50 281472585854032
x18 0x100080009 4295491593
x19 0x10 16
x20 0xfffc9c366130 281460417650992
x21 0x1 1
x22 0xffff70bbf000 281472573108224
x23 0x6ba19000 1805750272
x24 0x8 8
x25 0xffff70bbe000 281472573104128
x26 0xfffcafffd990 281460749621648
x27 0xfffcafffd998 281460749621656
x28 0xfffbd4001c60 281457058651232
x29 0xfffcafffd900 281460749621504
x30 0xffff7084c3f8 281472569492472
sp 0xfffcafffd8e0 0xfffcafffd8e0
pc 0xffff7084c404 0xffff7084c404 hcomm::HostCpuUrmaChannel::ParseInputParam()+308
cpsr 0x40001000 [ EL=0 BTYPE=0 SSBS Z ]
fpsr 0x0 [ ]
fpcr 0x0 [ Len=0 Stride=0 RMode=0 ]
vg 0x4 4
tpidr 0xfffcaffff8e0 0xfffcaffff8e0
tpidr2 0x0 0x0