[Bug]:PD场景：主meta和P实例同时被干掉后，备meta切换为主后，有大量日志刷屏-memcache-AtomGit

已关闭

[Bug]:PD场景：主meta和P实例同时被干掉后，备meta切换为主后，有大量日志刷屏 #330

hude01创建于 7 天前关闭于 4 天前

hude01成员

7 天前创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

API 令牌或密钥
密码或身份验证凭证
私有网址或接口地址
个人或机密数据
...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

- 操作系统  openEuler 24.03 (LTS-SP3)
- 昇腾硬件信息 hdk-25.5.1
- CANN软件版本 cann-9.1.0
- 安装的对应软件版本
memcache：release/1.2.1      commit id：9abec55b8b1579d3d564c2bff2e0b53777970ece
memfabric：release/1.2.1     commit id：7f37d0bb29db17734c0c5df47edec942d6aeaac6

🐛 问题描述

k8s + vllm + memcache池化（主备meta service） + dsv4 flash + PD场景：压测一段时间后，将主meta service和P实例同时kill后，备meta切换为主成功，但mmc-meta日志有刷屏现象
测试步骤：

1、将主备meta和PD拉起

2、通过ais_bench压测一段时间

3、将主meta service和P实例同时kill

预期结果：
备meta切换为主成功，且日志信息正常
实际结果：
备meta切换为主成功，mmc-meta日志有"warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2"刷屏现象

2026-09-16 20:06:03.244206 1471 info [AccLink acc_http_listener.cpp:56 ProcessNewConnection] Connected from 127.0.0.1:56266 successfully, ssl disable
2026-09-16 20:06:03.244474 1469 info [AccLink acc_http_link_default.cpp:131 AppendRecvData] Connection closed by peer on link [id:6130,fd:20,127.0.0.1:56266]
2026-09-16 20:06:03.897434 1339 warning [MMC mmc_leader.cpp:26 mmc_logger] Updated label of self.pod_name='meta-service-pod-1' to labels={'role': 'backup'}
2026-09-16 20:06:06.910690 1339 trace [MMC mmc_leader_election.cpp:158 CheckLeaderStatus] Pod meta-service-pod-1 became the leader
2026-09-16 20:06:06.934334 1339 warning [MMC mmc_leader.cpp:26 mmc_logger] Updated label of self.pod_name='meta-service-pod-1' to labels={'role': 'master'}
2026-09-16 20:06:07.190165 1346 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:40629 successfully, ssl disable
2026-09-16 20:06:07.190240 1346 info [MMC mmc_net_engine_acc.cpp:475 HandleNewLink] HandleNewLink with ignored rank, linkId: 6131
2026-09-16 20:06:07.190244 1346 trace [MMC mmc_meta_net_server.cpp:164 HandleNewLink] meta_service_MetaServer new link, id: -1
2026-09-16 20:06:07.190379 1346 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:40397 successfully, ssl disable
2026-09-16 20:06:07.190384 1346 info [MMC mmc_net_engine_acc.cpp:475 HandleNewLink] HandleNewLink with ignored rank, linkId: 6132
2026-09-16 20:06:07.190386 1346 trace [MMC mmc_meta_net_server.cpp:164 HandleNewLink] meta_service_MetaServer new link, id: -1
2026-09-16 20:06:07.190449 1346 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:51007 successfully, ssl disable
2026-09-16 20:06:07.190453 1346 info [MMC mmc_net_engine_acc.cpp:475 HandleNewLink] HandleNewLink with ignored rank, linkId: 6133
2026-09-16 20:06:07.190455 1346 trace [MMC mmc_meta_net_server.cpp:164 HandleNewLink] meta_service_MetaServer new link, id: -1
2026-09-16 20:06:07.190509 1346 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:1976 successfully, ssl disable
2026-09-16 20:06:07.190514 1346 info [MMC mmc_net_engine_acc.cpp:479 HandleNewLink] HandleNewLink with peer rankId: 9, linkId: 6134
2026-09-16 20:06:07.190516 1346 trace [MMC mmc_meta_net_server.cpp:164 HandleNewLink] meta_service_MetaServer new link, id: 9
2026-09-16 20:06:07.190563 1346 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:29168 successfully, ssl disable
2026-09-16 20:06:07.190566 1346 info [MMC mmc_net_engine_acc.cpp:475 HandleNewLink] HandleNewLink with ignored rank, linkId: 6135
2026-09-16 20:06:07.190568 1346 trace [MMC mmc_meta_net_server.cpp:164 HandleNewLink] meta_service_MetaServer new link, id: -1
2026-09-16 20:06:07.190613 1346 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:40520 successfully, ssl disable
2026-09-16 20:06:07.190617 1346 info [MMC mmc_net_engine_acc.cpp:479 HandleNewLink] HandleNewLink with peer rankId: 8, linkId: 6136
2026-09-16 20:06:07.190619 1346 trace [MMC mmc_meta_net_server.cpp:164 HandleNewLink] meta_service_MetaServer new link, id: 8
2026-09-16 20:06:07.190927 1463 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:38676 successfully, ssl disable
2026-09-16 20:06:07.191186 1463 trace [SMEM smem_tcp_config_store_server.cpp:271 LinkConnectedHandler] New link connected, linkId: 6137, worldSize: 1024, rankId: 12 reconnect:1
2026-09-16 20:06:07.191192 1463 trace [SMEM smem_tcp_config_store_server.cpp:276 LinkConnectedHandler] Success to fix world size:1024
2026-09-16 20:06:07.191195 1463 trace [SMEM smem_tcp_config_store_server.cpp:213 CanReceiveNewLink] change server state from INITED to RECOVER
2026-09-16 20:06:07.191293 1463 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:64613 successfully, ssl disable
2026-09-16 20:06:07.191297 1463 trace [SMEM smem_tcp_config_store_server.cpp:271 LinkConnectedHandler] New link connected, linkId: 6138, worldSize: 1024, rankId: 8 reconnect:1
2026-09-16 20:06:07.191369 1463 trace [AccLink acc_tcp_listener.cpp:210 ProcessNewConnection] Connected from 61.47.16.107:37688 successfully, ssl disable
2026-09-16 20:06:07.191373 1463 trace [SMEM smem_tcp_config_store_server.cpp:271 LinkConnectedHandler] New link connected, linkId: 6139, worldSize: 1024, rankId: 10 reconnect:1
2026-09-16 20:06:07.236045 1347 trace [MMC mmc_global_allocator.h:169 Mount] Mount bm on loc{rank=8,media=DRAM}, capacity:32212254720  successfully
2026-09-16 20:06:07.236137 1347 trace [MMC mmc_meta_manager.cpp:1277 Mount] SSD storage enabled for rank=8
2026-09-16 20:06:07.236147 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236150 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236152 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236154 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236155 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236157 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236159 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236161 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236163 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236165 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2
2026-09-16 20:06:07.236167 1347 warning [MMC mmc_blob_allocator.cpp:168 BuildFromBlobs] rebuild blob not match, allocator mediaType: DRAM, blob mediaType: 2

欢迎加入社区，感谢您对社区的贡献 🎉!

H hude01成员

7 天前添加了label：bug

shilinlee成员

7 天前评论：

/label add triaged

shilinlee成员

7 天前评论：

Issue 标签使用规范

请根据以下规则为 Issue 打标签，保持分类准确：

#	场景	标签
1	提交者本人提的 Issue	`/label add triaged`
2	属于 v1.0.0/release 里程碑的问题	`high-priority`
3	提交者本人正在跟进 / 排查中	`triage-review`
4	系统自动打标（无需手动操作）	`triaged`
5	测试同学提的 Issue	需确保复现步骤明确、所属仓库准确
6	开发同学提的建设性 Issue	按类型选择：`feature` / `rfc` / `performance`
7	已定位、待合入或待测试	`ok-to-test`
8	方案未明确、需团队讨论	`help-wanted`

⚠️ 测试or用户提 Issue 前请检查：问题描述是否清晰、仓库是否正确，避免信息不全影响处理效率。

ascend-robot成员

7 天前添加了label：triaged

Guoy_wang成员

4 天前将 guoywang 设为负责人

Guoy_wang成员

4 天前关联了pull request：fix: 聚合BM重注册时BuildFromBlobs的per-blob日志，修复meta日志刷屏

Z zhangjinshi成员

4 天前关闭了 issue

Z zhangjinshi成员

4 天前 issue状态由 TODO 改变为 DONE

ascend-robot成员

4 天前添加了label：resolved