已关闭
[Bug-Report|缺陷反馈]: msprobe对比图时打印报错:An sqlite3 error occurred: UNIQUE constraint failed: tb_nodes.id #132
menogrey创建于  2025年12月3日关闭于  2025年12月16日
menogrey
2025年12月3日 创建

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

执行命令msprobe -f pytorch graph -i ./compare.json -o ./output
除第一个step success外,后面的step全都打印error。

2025-12-03 01:23:45 (248855) [INFO] Start processing data for step0...
2025-12-03 01:23:46 (248923) [INFO] Start building graph for rank0...
2025-12-03 01:23:46 (248923) [INFO] Start building model graphs...
2025-12-03 01:23:46 (248925) [INFO] Start building graph for rank1...
2025-12-03 01:23:46 (248925) [INFO] Start building model graphs...
2025-12-03 01:23:46 (248927) [INFO] Start building graph for rank2...
2025-12-03 01:23:46 (248927) [INFO] Start building model graphs...
2025-12-03 01:23:46 (248929) [INFO] Start building graph for rank3...
2025-12-03 01:23:46 (248929) [INFO] Start building model graphs...
2025-12-03 01:23:46 (248931) [INFO] Start building graph for rank0...
2025-12-03 01:23:46 (248931) [INFO] Start building model graphs...
2025-12-03 01:23:46 (248929) [INFO] Building graph for step: step0, rank: rank3 finished.
2025-12-03 01:23:46 (248931) [INFO] Building graph for step: step0, rank: rank0 finished.
2025-12-03 01:23:46 (248925) [INFO] Building graph for step: step0, rank: rank1 finished.
2025-12-03 01:23:46 (248923) [INFO] Building graph for step: step0, rank: rank0 finished.
2025-12-03 01:23:46 (248927) [INFO] Building graph for step: step0, rank: rank2 finished.
2025-12-03 01:23:47 (248855) [INFO] [NPU] All tp groups is [[0, 1, 2, 3]].
2025-12-03 01:23:47 (248933) [INFO] Start comparing data for rank0...
2025-12-03 01:23:48 (248933) [INFO] Comparing data for rank0 finished.
2025-12-03 01:23:48 (248933) [INFO] Start exporting compare graph result, file name: compare_20251203012345.vis.db...
2025-12-03 01:23:49 (248933) [INFO] Exporting compare graph result successfully, the result file is saved in ./output2/compare_20251203012345.vis.db
2025-12-03 01:23:49 (248855) [INFO] Successfully exported compare graph results.
2025-12-03 01:23:49 (248855) [INFO] Start processing data for step1...
2025-12-03 01:23:50 (249092) [INFO] Start building graph for rank0...
2025-12-03 01:23:50 (249092) [INFO] Start building model graphs...
2025-12-03 01:23:50 (249094) [INFO] Start building graph for rank1...
2025-12-03 01:23:50 (249094) [INFO] Start building model graphs...
2025-12-03 01:23:50 (249096) [INFO] Start building graph for rank2...
2025-12-03 01:23:50 (249096) [INFO] Start building model graphs...
2025-12-03 01:23:50 (249098) [INFO] Start building graph for rank3...
2025-12-03 01:23:50 (249098) [INFO] Start building model graphs...
2025-12-03 01:23:50 (249100) [INFO] Start building graph for rank0...
2025-12-03 01:23:50 (249100) [INFO] Start building model graphs...
2025-12-03 01:23:51 (249092) [INFO] Building graph for step: step1, rank: rank0 finished.
2025-12-03 01:23:51 (249096) [INFO] Building graph for step: step1, rank: rank2 finished.
2025-12-03 01:23:51 (249100) [INFO] Building graph for step: step1, rank: rank0 finished.
2025-12-03 01:23:51 (249098) [INFO] Building graph for step: step1, rank: rank3 finished.
2025-12-03 01:23:51 (249094) [INFO] Building graph for step: step1, rank: rank1 finished.
2025-12-03 01:23:51 (248855) [INFO] [NPU] All tp groups is [[0, 1, 2, 3]].
2025-12-03 01:23:51 (249102) [INFO] Start comparing data for rank0...
2025-12-03 01:23:52 (249102) [INFO] Comparing data for rank0 finished.
2025-12-03 01:23:52 (249102) [INFO] Start exporting compare graph result, file name: compare_20251203012345.vis.db...
2025-12-03 01:23:53 (249102) [ERROR] An sqlite3 error occurred: UNIQUE constraint failed: tb_nodes.id
2025-12-03 01:23:53 (249102) [ERROR] Failed to export compare graph result, file: compare_20251203012345.vis.db, error: An sqlite3 error occurred
2025-12-03 01:23:53 (248855) [ERROR] Unable to export compare graph results: compare_20251203012345.vis.db.
2025-12-03 01:23:53 (248855) [INFO] Start processing data for step10...
2025-12-03 01:23:54 (249258) [INFO] Start building graph for rank0...
2025-12-03 01:23:54 (249260) [INFO] Start building graph for rank1...
2025-12-03 01:23:54 (249258) [INFO] Start building model graphs...
2025-12-03 01:23:54 (249262) [INFO] Start building graph for rank2...
2025-12-03 01:23:54 (249260) [INFO] Start building model graphs...
2025-12-03 01:23:54 (249262) [INFO] Start building model graphs...
2025-12-03 01:23:54 (249264) [INFO] Start building graph for rank3...
2025-12-03 01:23:54 (249264) [INFO] Start building model graphs...
2025-12-03 01:23:54 (249266) [INFO] Start building graph for rank0...
2025-12-03 01:23:54 (249266) [INFO] Start building model graphs...
2025-12-03 01:23:55 (249258) [INFO] Building graph for step: step10, rank: rank0 finished.
2025-12-03 01:23:55 (249264) [INFO] Building graph for step: step10, rank: rank3 finished.
2025-12-03 01:23:55 (249262) [INFO] Building graph for step: step10, rank: rank2 finished.
2025-12-03 01:23:55 (249260) [INFO] Building graph for step: step10, rank: rank1 finished.
2025-12-03 01:23:55 (249266) [INFO] Building graph for step: step10, rank: rank0 finished.
2025-12-03 01:23:55 (248855) [INFO] [NPU] All tp groups is [[0, 1, 2, 3]].
2025-12-03 01:23:56 (249268) [INFO] Start comparing data for rank0...
2025-12-03 01:23:57 (249268) [INFO] Comparing data for rank0 finished.
2025-12-03 01:23:57 (249268) [INFO] Start exporting compare graph result, file name: compare_20251203012345.vis.db...
2025-12-03 01:23:57 (249268) [ERROR] An sqlite3 error occurred: UNIQUE constraint failed: tb_nodes.id
2025-12-03 01:23:57 (249268) [ERROR] Failed to export compare graph result, file: compare_20251203012345.vis.db, error: An sqlite3 error occurred
2025-12-03 01:23:57 (248855) [ERROR] Unable to export compare graph results: compare_20251203012345.vis.db.
2025-12-03 01:23:58 (248855) [INFO] Start processing data for step11...
2025-12-03 01:23:59 (249424) [INFO] Start building graph for rank0...
2025-12-03 01:23:59 (249424) [INFO] Start building model graphs...
2025-12-03 01:23:59 (249426) [INFO] Start building graph for rank1...
2025-12-03 01:23:59 (249426) [INFO] Start building model graphs...
2025-12-03 01:23:59 (249428) [INFO] Start building graph for rank2...
2025-12-03 01:23:59 (249428) [INFO] Start building model graphs...
2025-12-03 01:23:59 (249430) [INFO] Start building graph for rank3...
2025-12-03 01:23:59 (249430) [INFO] Start building model graphs...
2025-12-03 01:23:59 (249432) [INFO] Start building graph for rank0...
2025-12-03 01:23:59 (249432) [INFO] Start building model graphs...
2025-12-03 01:23:59 (249424) [INFO] Building graph for step: step11, rank: rank0 finished.
2025-12-03 01:23:59 (249426) [INFO] Building graph for step: step11, rank: rank1 finished.
2025-12-03 01:23:59 (249432) [INFO] Building graph for step: step11, rank: rank0 finished.
2025-12-03 01:23:59 (249428) [INFO] Building graph for step: step11, rank: rank2 finished.
2025-12-03 01:23:59 (249430) [INFO] Building graph for step: step11, rank: rank3 finished.
2025-12-03 01:24:00 (248855) [INFO] [NPU] All tp groups is [[0, 1, 2, 3]].
2025-12-03 01:24:00 (249434) [INFO] Start comparing data for rank0...
2025-12-03 01:24:01 (249434) [INFO] Comparing data for rank0 finished.
2025-12-03 01:24:01 (249434) [INFO] Start exporting compare graph result, file name: compare_20251203012345.vis.db...
2025-12-03 01:24:01 (249434) [ERROR] An sqlite3 error occurred: UNIQUE constraint failed: tb_nodes.id
2025-12-03 01:24:01 (249434) [ERROR] Failed to export compare graph result, file: compare_20251203012345.vis.db, error: An sqlite3 error occurred
2025-12-03 01:24:01 (248855) [ERROR] Unable to export compare graph results: compare_20251203012345.vis.db.

Environment / 环境信息 (Mandatory / 必填)

910c

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

对比开tp和不开tp的vllm-ascend dump信息,参考 https://docs.vllm.ai/projects/ascend/en/latest/developer_guide/performance_and_debug/msprobe_guide.html#id1

Describe the expected behavior / 预期结果 (Mandatory / 必填)

正确执行

上面已提供

Special notes for this issue/备注 (Optional / 选填)

likedislike
louyujing成员
2025年12月3日 评论:

【问题根因】
当前不同切分合并比对不支持step批量比对,仅支持单个step数据之间的比对
【解决方案】
将支持不同切分合并批量step比对
【问题状态】
修复中

likedislike
louyujing成员
2025年12月5日 评论:

【问题根因】
当前不同切分合并比对不支持step批量比对,仅支持单个step数据之间的比对
【解决方案】
将支持不同切分合并批量step比对
【问题状态】
已修复,pr: https://gitcode.com/Ascend/mstt/pull/5734,请拉取master分支最新代码编包使用

likedislike
Llouyujing成员
2025年12月5日 将 louyujing 设为负责人
yangchenyangchen成员
2025年12月16日 issue状态由 TODO 改变为 DONE
yangchenyangchen成员
2025年12月16日 关闭了 issue