Star138
127
代码
代码
Issues2
Pull Requests3
项目讨论
Wiki
分析
Star138
127
已关闭
[Bug-Report|缺陷反馈]: 在最新的CANN 8.3 RC1+torchnpu 7.2+torch2.8下分析多卡场景报错 #135
布衣创建于  2025年12月27日关闭于  2025年12月29日
布衣
布衣
2025年12月27日 创建

Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

在单机双卡上使用profiler采集性能数据后使用advisor进行分析，遇到报错：
命令为：msprof-analyze advisor all -d ./prof_result -o ./advisor_output
报错误：

(torch) root:~/work/filestorage/zhoufei/hermite/my-ultra# msprof-analyze advisor all -d ./prof_result -o ./advisor_output
[2025-12-27 21:16:40][WARNING] Security Warning: Do not run this tool as root. Running with elevated privileges may compromise system security. Use a regular user account.
[2025-12-27 21:16:41][INFO] cluster analysis is in the process, please wait...
[2025-12-27 21:16:41][INFO] Begin generate communication data.
[2025-12-27 21:16:41][INFO] Recipe CommunicationGroupMap analysis is starting to launch.
[2025-12-27 21:16:41][INFO] context concurrent initialized.
[2025-12-27 21:16:41][INFO] CommunicationGroupMap init.
[2025-12-27 21:16:42][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152038_20251227211315187_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:42][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152039_20251227211315188_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:42][WARNING] No data to dump, skipping.
[2025-12-27 21:16:43][INFO] Recipe CommunicationGroupMap analysis launched successfully.
[2025-12-27 21:16:43][INFO] Recipe CommunicationTimeSum analysis is starting to launch.
[2025-12-27 21:16:43][INFO] context concurrent initialized.
[2025-12-27 21:16:43][INFO] CommunicationTimeSum init.
[2025-12-27 21:16:43][INFO] Run CommunicationGroupMap recipe first to get CommunicationGroupMapping table
[2025-12-27 21:16:43][INFO] CommunicationGroupMap init.
[2025-12-27 21:16:44][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152038_20251227211315187_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:44][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152039_20251227211315188_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:44][WARNING] No data to dump, skipping.
[2025-12-27 21:16:44][ERROR] Create CommunicationGroupMap table failed! Skip CommunicationTimeSum.
[2025-12-27 21:16:44][INFO] Recipe CommunicationTimeSum analysis launched successfully.
[2025-12-27 21:16:44][INFO] Recipe CommMatrixSum analysis is starting to launch.
[2025-12-27 21:16:44][INFO] context concurrent initialized.
[2025-12-27 21:16:44][INFO] CommMatrixSum init.
[2025-12-27 21:16:45][WARNING] This table CommAnalyzerMatrix does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152038_20251227211315187_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:45][WARNING] This table CommAnalyzerMatrix does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152039_20251227211315188_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:45][ERROR] Communication matrix data is None.
[2025-12-27 21:16:45][WARNING] No data to dump, skipping.
[2025-12-27 21:16:46][INFO] Recipe CommMatrixSum analysis launched successfully.
Cluster analyzing:   0%|                                                                                                                                                                                                                                           | 0/3[2025-12-27 21:16:47][WARNING] ClusterBaseInfoAnalysis skipped, since no metadata or distributed args found
Cluster analyzing:   0%|                                                                                                                                                                                                                                           | 0/3[2025-12-27 21:16:47][WARNING] This table CommunicationGroupMapping does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/advisor_output/cluster_analysis_output/cluster_analysis.db.
[2025-12-27 21:16:47][ERROR] There is no CommunicationGroupMapping data in /root/work/filestorage/zhoufei/hermite/my-ultra/advisor_output/cluster_analysis_output/cluster_analysis.db.
[2025-12-27 21:16:47][INFO] HostInfoAnalysis completed
Cluster analyzing:  33%|██████████████████████████████████████████████████████████████████████████████                                                                                                                                                             | 1/3[2025-12-27 21:16:47][INFO] StepTraceTimeAnalysis completed
Cluster analyzing:  67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                               | 2/3
[2025-12-27 21:16:47][INFO] The cluster analysis result file has been generated: /root/work/filestorage/zhoufei/hermite/my-ultra/advisor_output
[2025-12-27 21:16:47][INFO] Cluster has been analyzed because of the existence of cluster analysis output directory.
[2025-12-27 21:16:47][INFO] Skip Cluster analyze backend.
[2025-12-27 21:16:47][ERROR] File ./advisor_output/cluster_analysis_output/cluster_analysis.db read failed error: Execution failed on sql '
SELECT
    step,
    rank_id,
    band_type,
    transit_time,
    transit_size
FROM ClusterCommAnalyzerBandwidth
WHERE band_type IN ('SDMA', 'RDMA')
': no such table: ClusterCommAnalyzerBandwidth
[2025-12-27 21:16:47][ERROR] 'NoneType' object is not subscriptable
[2025-12-27 21:16:47][ERROR] 'NoneType' object has no attribute 'get_data'
[2025-12-27 21:16:47][ERROR] Do analysis error.

单卡分析则正常通过（虽然分析时advisor报出的环境是cann8.0+torch2.1，这是错误的）

Environment / 环境信息 (Mandatory / 必填)

4U鲲鹏920+8卡910b2 ubuntu 2004的k8s容器环境

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

profiler配置：

        profile = getattr(self.args, "training_profile", False)
        if profile:
            try:
                import torch_npu
            except:
                LOGGER.warning(colorstr("red", "NPU-未安装torch_npu，请检查NPU环境"))
                setattr(self.args, "training_profile", False)
            else:
                experimental_config = torch_npu.profiler._ExperimentalConfig(
                    export_type=[torch_npu.profiler.ExportType.Text],
                    profiler_level=torch_npu.profiler.ProfilerLevel.Level0,
                    mstx=False,  # 原参数名msprof_tx改为mstx，新版本依旧兼容原参数名msprof_tx
                    aic_metrics=torch_npu.profiler.AiCMetrics.AiCoreNone,
                    l2_cache=False,
                    op_attr=False,
                    data_simplification=False,
                    record_op_args=False,
                    gc_detect_threshold=None,
                )
                detailed = profile == "detailed"
                prof = torch_npu.profiler.profile(
                    activities=[torch_npu.profiler.ProfilerActivity.CPU, torch_npu.profiler.ProfilerActivity.NPU],
                    schedule=torch_npu.profiler.schedule(
                        wait=0,
                        warmup=5,
                        active=2,
                        repeat=1,
                        skip_first=20,
                    ),  # 与prof.step()配套使用
                    on_trace_ready=torch_npu.profiler.tensorboard_trace_handler("./prof_result"),
                    record_shapes=detailed,
                    profile_memory=detailed,
                    with_stack=detailed,
                    with_modules=detailed,
                    with_flops=detailed,
                    experimental_config=experimental_config,
                )
                self.prof = prof
                LOGGER.info(colorstr("NPU-性能分析准备"))

采集模式设定为detailed与否均报此错误。

分析命令：

msprof-analyze advisor all -d ./prof_result -o ./advisor_output

Describe the expected behavior / 预期结果 (Mandatory / 必填)

输出分析结果

(torch) root:~/work/filestorage/zhoufei/hermite/my-ultra# msprof-analyze advisor all -d ./prof_result -o ./advisor_output
[2025-12-27 21:16:40][WARNING] Security Warning: Do not run this tool as root. Running with elevated privileges may compromise system security. Use a regular user account.
[2025-12-27 21:16:41][INFO] cluster analysis is in the process, please wait...
[2025-12-27 21:16:41][INFO] Begin generate communication data.
[2025-12-27 21:16:41][INFO] Recipe CommunicationGroupMap analysis is starting to launch.
[2025-12-27 21:16:41][INFO] context concurrent initialized.
[2025-12-27 21:16:41][INFO] CommunicationGroupMap init.
[2025-12-27 21:16:42][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152038_20251227211315187_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:42][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152039_20251227211315188_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:42][WARNING] No data to dump, skipping.
[2025-12-27 21:16:43][INFO] Recipe CommunicationGroupMap analysis launched successfully.
[2025-12-27 21:16:43][INFO] Recipe CommunicationTimeSum analysis is starting to launch.
[2025-12-27 21:16:43][INFO] context concurrent initialized.
[2025-12-27 21:16:43][INFO] CommunicationTimeSum init.
[2025-12-27 21:16:43][INFO] Run CommunicationGroupMap recipe first to get CommunicationGroupMapping table
[2025-12-27 21:16:43][INFO] CommunicationGroupMap init.
[2025-12-27 21:16:44][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152038_20251227211315187_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:44][WARNING] This table CommAnalyzerTime does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152039_20251227211315188_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:44][WARNING] No data to dump, skipping.
[2025-12-27 21:16:44][ERROR] Create CommunicationGroupMap table failed! Skip CommunicationTimeSum.
[2025-12-27 21:16:44][INFO] Recipe CommunicationTimeSum analysis launched successfully.
[2025-12-27 21:16:44][INFO] Recipe CommMatrixSum analysis is starting to launch.
[2025-12-27 21:16:44][INFO] context concurrent initialized.
[2025-12-27 21:16:44][INFO] CommMatrixSum init.
[2025-12-27 21:16:45][WARNING] This table CommAnalyzerMatrix does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152038_20251227211315187_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:45][WARNING] This table CommAnalyzerMatrix does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/prof_result/dl-9078e29da8f63b5aacbb340a12187fb9-r0-dfe4fc78fe7d-0_152039_20251227211315188_ascend_pt/ASCEND_PROFILER_OUTPUT/analysis.db.
[2025-12-27 21:16:45][ERROR] Communication matrix data is None.
[2025-12-27 21:16:45][WARNING] No data to dump, skipping.
[2025-12-27 21:16:46][INFO] Recipe CommMatrixSum analysis launched successfully.
Cluster analyzing:   0%|                                                                                                                                                                                                                                           | 0/3[2025-12-27 21:16:47][WARNING] ClusterBaseInfoAnalysis skipped, since no metadata or distributed args found
Cluster analyzing:   0%|                                                                                                                                                                                                                                           | 0/3[2025-12-27 21:16:47][WARNING] This table CommunicationGroupMapping does not exist in this database /root/work/filestorage/zhoufei/hermite/my-ultra/advisor_output/cluster_analysis_output/cluster_analysis.db.
[2025-12-27 21:16:47][ERROR] There is no CommunicationGroupMapping data in /root/work/filestorage/zhoufei/hermite/my-ultra/advisor_output/cluster_analysis_output/cluster_analysis.db.
[2025-12-27 21:16:47][INFO] HostInfoAnalysis completed
Cluster analyzing:  33%|██████████████████████████████████████████████████████████████████████████████                                                                                                                                                             | 1/3[2025-12-27 21:16:47][INFO] StepTraceTimeAnalysis completed
Cluster analyzing:  67%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                                                               | 2/3
[2025-12-27 21:16:47][INFO] The cluster analysis result file has been generated: /root/work/filestorage/zhoufei/hermite/my-ultra/advisor_output
[2025-12-27 21:16:47][INFO] Cluster has been analyzed because of the existence of cluster analysis output directory.
[2025-12-27 21:16:47][INFO] Skip Cluster analyze backend.
[2025-12-27 21:16:47][ERROR] File ./advisor_output/cluster_analysis_output/cluster_analysis.db read failed error: Execution failed on sql '
SELECT
    step,
    rank_id,
    band_type,
    transit_time,
    transit_size
FROM ClusterCommAnalyzerBandwidth
WHERE band_type IN ('SDMA', 'RDMA')
': no such table: ClusterCommAnalyzerBandwidth
[2025-12-27 21:16:47][ERROR] 'NoneType' object is not subscriptable
[2025-12-27 21:16:47][ERROR] 'NoneType' object has no attribute 'get_data'
[2025-12-27 21:16:47][ERROR] Do analysis error.

Special notes for this issue/备注 (Optional / 选填)

likedislike
fanglanyue
fanglanyue成员
2025年12月29日 评论：

level0下默认不采集通信小算子，因此在分析时报没有通信带宽bandwidth，通信矩阵matrix的错误。推荐使用level1及以上。感谢您的反馈，相关报错信息我们将做优化。
image.png
image.png

likedislike
布衣
布衣
2025年12月29日 评论：

的确如此，问题已得到解决，非常感谢

likedislike
布衣布衣
2025年12月29日 issue状态由 TODO 改变为 DONE
布衣布衣
2025年12月29日 关闭了 issue