已关闭
[Bug]: query_dop=3 执行列存 HashJoin 查询时 DN crash (connection lost),DN 日志 Disk Spilled #8364
wuchenlong创建于  7月30日关闭于  17 天前
wuchenlong
wuchenlong成员
7月30日 创建

测试类型

功能测试

测试版本

openGauss 7.0.0-RC3 build 38d194a0 (B015)

问题描述

设置 query_dop=3 后,执行带 HashJoin 的列存表复杂查询时,gsql 立即报错 connection to server was lost,主节点 Primary Down。DN 日志显示 VecHashJoin 发生 Disk Spilled (early spilled),随后 fatal 退出。query_dop=1 时相同查询正常执行。

已按文档《高级特性-并行查询》的配置步骤完整验证:系统负载确认充足(CPU 空闲 94.5%)、dop=1 EXPLAIN 确认计划适用并行、dop=3 EXPLAIN 确认并行计划已生成、执行查询即 crash。

操作系统和硬件信息

openEuler 24.03 LTS / aarch64 (Kunpeng 920) / 8 核 / 30 GB 内存 / 3节点 1主2备部署

测试环境

本地 3 节点 openGauss 7.0.0-RC3 (B015) 主备集群,DN 端口 15600,节点 20.20.20.224(主) / 20.20.20.226(备) / 20.20.20.228(备)。work_mem=64MB, shared_buffers=128MB。

被测功能

并行查询 (query_dop + SMP)

预置条件

数据库正常运行(cluster_state: Normal)

操作步骤

【第一步:观察系统负载 — 确认资源充足,符合文档要求(<50%)】

top - 15:08:00 up 11 days, load average: 1.68
%Cpu(s):  2.7 us, 94.5 id   ← CPU 空闲 94.5%,远低于 50% 阈值
MiB Mem: 30990.0 total, ~13GB avail Mem

结论:资源充足,适合使用 SMP 并行。

【第二步:创建列存表 + 100万行数据 + query_dop=1 EXPLAIN 验证计划符合并行场景】

drop table if exists column_table;
create table column_table (id int, a int, b char(20)) with (orientation = column);
drop table if exists column_table_01;
create table column_table_01 (id int, a int, b char(20)) with (orientation = column);

insert into column_table select id, id % 10, 'foo' from generate_series(1, 1000000) id;
insert into column_table_01 select id, id % 10, 'foo' from generate_series(1, 1000000) id;

analyze column_table;
analyze column_table_01;

SET query_dop = 1;
EXPLAIN (COSTS OFF)
select table_002.id column_006, table_002.b column_007,
       table_003.column_004 column_008, table_003.column_005 column_009
from column_table table_002
inner join (
    select table_001.id column_001, table_001.a column_002,
           table_002.a column_003, table_002.b column_004,
           table_001.b column_005
    from column_table table_002
    full outer join column_table_01 table_001
    on table_002.a = table_001.id
) table_003 on table_002.a = table_003.column_001
limit 10;

输出:计划含 Vector HashJoin + CStore Scan → 符合文档并行适用场景。

【第三步:query_dop=3 EXPLAIN 验证并行计划已生成】

SET query_dop = 3;
EXPLAIN (COSTS OFF) -- 同上查询

输出:

 id |                                  operation
----+------------------------------------------------------------------------------
  1 | ->  Row Adapter
  2 |    ->  Vector Limit
  3 |       ->  Vector Streaming(type: LOCAL GATHER dop: 1/3)
  4 |          ->  Vector Limit
  5 |             ->  Vector Sonic Hash Join (6,12)
  6 |                ->  Vector Streaming(type: BROADCAST dop: 3/3)
  7 |                   ->  Vector Hash Right Join (8, 10)
  8 |                      ->  Vector Streaming(type: LOCAL REDISTRIBUTE dop: 3/3)
  9 |                         ->  CStore Scan on column_table table_002
 10 |                      ->  Vector Streaming(type: LOCAL REDISTRIBUTE dop: 3/3)
 11 |                         ->  CStore Scan on column_table_01 table_001
 12 |                ->  CStore Scan on column_table table_002

结论:LOCAL GATHER(dop 1/3)、LOCAL BROADCAST、LOCAL REDISTRIBUTE 均已出现,并行计划正确生成。

【第四步:执行相同查询(非 EXPLAIN)→ crash】

SET query_dop = 3;
-- 直接执行查询
select table_002.id column_006, table_002.b column_007,
       table_003.column_004 column_008, table_003.column_005 column_009
from column_table table_002
inner join (
    select table_001.id column_001, table_001.a column_002,
           table_002.a column_003, table_002.b column_004,
           table_001.b column_005
    from column_table table_002
    full outer join column_table_01 table_001
    on table_002.a = table_001.id
) table_003 on table_002.a = table_003.column_001
limit 10;

输出:

SET
connection to server was lost

【第五步:清理】

drop table if exists column_table;
drop table if exists column_table_01;

注:crash 后清理无法执行,需先 gs_om -t restart 恢复集群。

预期输出

query_dop=3 时查询正常返回 10 rows,不 crash

实际输出

gsql 输出: SET
connection to server was lost
主节点状态变为 Primary Down,后续连接失败: failed to connect /home/wcl0623/cluster/tmp:15600

日志信息

DN pg_log 关键行(dn_6001 postgresql-*.log):

  • VecHashJoin(7) Disk Spilled
  • Memory is not enough, early spilled
  • FATAL: server process exited abnormally

crash 后需手动 gs_om -t restart 恢复集群

提单组织

测试团队

测试代码

Yat GUC 测试套: Opengauss_Function_Tools_Query_Dop_Case0038.py
路径: Yat/openGaussBase/testcase/GUC/VPC/

likedislike
wuchenlongwuchenlong成员
7月30日 添加了label:bug
opengauss_bot
opengauss_bot成员
7月30日 评论:

This issue requires an assignee. Since you haven't specified one, we've assigned TestManager as the default assignee for this issue.

likedislike
opengauss_botopengauss_bot成员
7月30日 将 TestManager 设为负责人
opengauss_botopengauss_bot成员
7月30日 添加了label:sig/StorageEngine,sig/AI,sig/CM,sig/CloudNative,sig/SecurityTechnology,sig/SQLEngine
opengauss_bot
opengauss_bot成员
7月30日 评论:

Welcome To openGauss Community

Hey @wuchenlong , thanks for your contribution to the community.

Bot Usage Manual

I'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands.

Contact Guide

If you have any questions, please contact the SIG: StorageEngine, AI, CM, CloudNative, SecurityTechnology, SQLEngine ,
and any of the maintainers: @CarrotGo, @chendong76, @chenxiaobin19, @congzhou2603, @dodders, @hwworkholic, @jemappellehc, @muyulinzhong, @quemingjian, @shenzheng4, @shirley_zhengx, @superlchf, @totaj, @wlff234, @wofanzheng, @ywzq1161327784 ,
and any of the committers: @Igali, @bihua111, @cailei19, @h_ray, @levy53071, @libiao2024, @lihaixiao, @mrzack, @wangfeihuo, @wuyuechuan, @xiong_xjun, @zhangfengzhi123, @zhangxubo, @zhangzq131, @zjh_hw .

likedislike
wuchenlongwuchenlong成员
7月31日 关联了看板:临时看板
chendong76chendong76成员
8月3日 关联了看板:openGauss 7.0.0-LTS
chendong76chendong76成员
8月3日 移除了看板:临时看板
wuchenlongwuchenlong成员
8月3日 issue优先级由 无优先级 改变为 次要
ywzq1161327784ywzq1161327784成员
8月4日 移除了负责人 TestManager
opengauss_bot
opengauss_bot成员
8月4日 评论:

This issue requires an assignee. Since you haven't specified one, we've assigned TestManager as the default assignee for this issue.

likedislike
opengauss_botopengauss_bot成员
8月4日 将 TestManager 设为负责人
ywzq1161327784ywzq1161327784成员
8月4日 将 superlchf 设为负责人,移除负责人 TestManager
ywzq1161327784ywzq1161327784成员
8月4日 移除了负责人 superlchf
opengauss_bot
opengauss_bot成员
8月4日 评论:

This issue requires an assignee. Since you haven't specified one, we've assigned TestManager as the default assignee for this issue.

likedislike
opengauss_botopengauss_bot成员
8月4日 将 TestManager 设为负责人
ywzq1161327784ywzq1161327784成员
8月4日 将 wofanzheng 设为负责人,移除负责人 TestManager
wofanzhengwofanzheng成员
8月18日 关联了pull request:修复本地并行流完成消息导致的崩溃
wofanzhengwofanzheng成员
29 天前 issue状态由 待办的 改变为 已完成
wofanzheng
wofanzheng成员
22 天前 评论:

自验证OK:

DROP TABLE IF EXISTS column_table;
CREATE TABLE column_table (
    id int,
    a int,
    b char(20)
) WITH (orientation = column);

DROP TABLE IF EXISTS column_table_01;
CREATE TABLE column_table_01 (
    id int,
    a int,
    b char(20)
) WITH (orientation = column);

INSERT INTO column_table
SELECT id, id % 10, 'foo'
FROM generate_series(1, 1000000) id;

INSERT INTO column_table_01
SELECT id, id % 10, 'foo'
FROM generate_series(1, 1000000) id;

ANALYZE column_table;
ANALYZE column_table_01;

SET work_mem = '64MB';

-- 对照:dop=1 正常返回 10 行
SET query_dop = 1;

SELECT table_002.id column_006,
       table_002.b column_007,
       table_003.column_004 column_008,
       table_003.column_005 column_009
FROM column_table table_002
INNER JOIN (
    SELECT table_001.id column_001,
           table_001.a column_002,
           table_002.a column_003,
           table_002.b column_004,
           table_001.b column_005
    FROM column_table table_002
    FULL OUTER JOIN column_table_01 table_001
    ON table_002.a = table_001.id
) table_003
ON table_002.a = table_003.column_001
LIMIT 10;

-- 先确认 dop=3 会生成并行计划,能看到 LOCAL GATHER / BROADCAST / REDISTRIBUTE
SET query_dop = 3;

EXPLAIN (COSTS OFF)
SELECT table_002.id column_006,
       table_002.b column_007,
       table_003.column_004 column_008,
       table_003.column_005 column_009
FROM column_table table_002
INNER JOIN (
    SELECT table_001.id column_001,
           table_001.a column_002,
           table_002.a column_003,
           table_002.b column_004,
           table_001.b column_005
    FROM column_table table_002
    FULL OUTER JOIN column_table_01 table_001
    ON table_002.a = table_001.id
) table_003
ON table_002.a = table_003.column_001
LIMIT 10;

SELECT table_002.id column_006,
       table_002.b column_007,
       table_003.column_004 column_008,
       table_003.column_005 column_009
FROM column_table table_002
INNER JOIN (
    SELECT table_001.id column_001,
           table_001.a column_002,
           table_002.a column_003,
           table_002.b column_004,
           table_001.b column_005
    FROM column_table table_002
    FULL OUTER JOIN column_table_01 table_001
    ON table_002.a = table_001.id
) table_003
ON table_002.a = table_003.column_001
LIMIT 10;

image.png

likedislike
wofanzhengwofanzheng成员
22 天前 issue状态由 已完成 改变为 待回归
wuchenlong
wuchenlong成员
21 天前 评论:

【回归版本】openGauss 7.0.0 build 93bfec54(7.0.0LTS B020,compiled at 2026-08-26 10:10:18)
【回归日期】20260827
【回归人员】wuchenlong
【回归结论】通过

测试结果

按 issue 原步骤 + 开发自验证 SQL,在 224 主备集群(wcl0623:15600,Primary=Testci044)执行:

  • 列存表 column_table / column_table_01 各 100 万行,work_mem=64MB
  • query_dop=1:正常返回 10 行
  • query_dop=3 EXPLAIN:计划含 LOCAL GATHER dop: 1/3BROADCAST dop: 3/3LOCAL REDISTRIBUTE dop: 3/3、Vector Sonic Hash Join
  • query_dop=3 执行:正常返回 10 行,无 connection to server was lost,集群保持 Normal

关联修复:openGauss-server !9598(DestBatchLocalBroadCast 误走 FE/libcomm 完成消息路径)。

likedislike
liuzhen001liuzhen001成员
17 天前 issue状态由 待回归 改变为 已验收
liuzhen001liuzhen001成员
17 天前 关闭了 issue