已关闭
[Bug]: CI用例清理表空间时core掉(偶现) #318
刘凯创建于  8月7日关闭于  8月19日
刘凯
刘凯成员
8月7日 创建

测试类型

存储功能

测试版本

7.0.0-RC3

问题描述

操作系统和硬件信息

NAME="openEuler"
VERSION="22.03 LTS"
ID="openEuler"
VERSION_ID="22.03"
PRETTY_NAME="openEuler 22.03 LTS"
ANSI_COLOR="0;31"

测试环境

oGRAC双节点

被测功能

预置条件

操作步骤

SQL> DROP TABLESPACE xxxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxxx INCLUDING CONTENTS AND DATAFILES;

OG-03200, cluster DDL operation not available, please check whether other instances are open

预期输出

Succeed

实际输出

OG-03200, cluster DDL operation not available, please check whether other instances are open

日志信息

Program terminated with signal SIGABRT, Aborted.
#0  0x0000ffff8d6e2ce0 in ?? () from /usr/lib64/libc.so.6
[Current thread is 1 (Thread 0xfffac8baf620 (LWP 3832741))]
(gdb) bt
#0  0x0000ffff8d6e2ce0 in ?? () from /usr/lib64/libc.so.6
#1  0x0000ffff8d69befc in raise () from /usr/lib64/libc.so.6
#2  0x0000ffff8d689d70 in abort () from /usr/lib64/libc.so.6
#3  0x0000aaaae1e47cf8 in proc_sign_func (sig_num=<optimized out>, siginfo=0xfffac8bad9b0, context=0xfffac8bada30)
    at /home/oGRAC/pkg/src/server/srv_blackbox.c:740
#4  <signal handler called>
#5  0x0000aaaae224c91c in cm_panic (condition=0) at /home/oGRAC/pkg/src/common/cm_debug.h:90
#6  buf_enter_page (mode=LATCH_MODE_S, options=0 '\000', page_id=..., session=0xfffad0f57000)
    at /home/oGRAC/pkg/src/kernel/buffer/knl_buffer_access.h:94
#7  buf_check_resident_page_version (session=session@entry=0xfffad0f57000, page_id=...) at /home/oGRAC/pkg/src/kernel/buffer/knl_buffer.c:1828
#8  0x0000aaaae22bfadc in rmon_free_spc_extents (rmon_ctx=0xffff4acf0c78, session=0xfffad0f57000)
    at /home/oGRAC/pkg/src/kernel/daemon/knl_rmon.c:220
#9  rmon_proc (thread=0xffff4acf0c78) at /home/oGRAC/pkg/src/kernel/daemon/knl_rmon.c:317
#10 0x0000ffff8d9aef30 in cm_thread_run (arg=<optimized out>) at /home/oGRAC/pkg/src/common/cm_thread.c:241
#11 0x0000ffff8d6e11e8 in ?? () from /usr/lib64/libc.so.6
#12 0x0000ffff8d7485dc in ?? () from /usr/lib64/libc.so.6
(gdb) 
UTC+08:00 2026-08-07 14:39:28.703|KNL_COMM|00000|3832735|INFO>start trigger full checkpoint, expect process pages 1 [knl_ckpt.c:530]
UTC+08:00 2026-08-07 14:39:28.710|KNL_COMM|00000|3832735|INFO>Finish trigger full checkpoint, Flush pages 1, Clean edp count 0, cost time(us) 6306 [knl_ckpt.c:606]
UTC+08:00 2026-08-07 14:39:28.710|CLUSTER|00000|3833524|INFO>[CKPT] done request to trigger checkpoint, type = 2 [dtc_ckpt.c:805]
UTC+08:00 2026-08-07 14:39:28.716|CLUSTER|00000|3833343|INFO>[DTC][dtc_process_remove_df_watch]: remove device watch for df 19 [dtc_dc.c:798]
UTC+08:00 2026-08-07 14:39:28.785|CLUSTER|00000|3833522|INFO>[DTC][dtc_process_remove_df_watch]: remove device watch for df 20 [dtc_dc.c:798]
UTC+08:00 2026-08-07 14:39:28.871|SPACE|00000|3833592|INFO>logic to remove space id is 13. [knl_space_log.c:219]
UTC+08:00 2026-08-07 14:39:28.871|KNL_COMM|00000|3833592|INFO>[CKPT] remove df page, count=0, df=+vg1/xxx. [knl_ckpt.c:3291]
UTC+08:00 2026-08-07 14:39:28.946|SPACE|00000|3833592|INFO>[SPACE] space TABSPACE_MPART_REBINDEX_NOENC_1 remove mount datafile  success [knl_drop_space.c:758]
UTC+08:00 2026-08-07 14:39:28.948|KNL_COMM|00000|3833592|INFO>[CKPT] remove df page, count=0, df=+vg1/xxx. [knl_ckpt.c:3291]
UTC+08:00 2026-08-07 14:39:29.003|BUFFER|00000|3832741|ERROR>[BUFFER] offlined tablespace 4294967295 or datafile of page_id 19-1 [knl_buffer_access.c:264]
UTC+08:00 2026-08-07 14:39:29.003|BUFFER|00000|3832741|ERROR>OG-00758:[BUFFER][19-1][buf load page] failed, load_status:2, lock_mode:2,The tablespace invalid space is offline, buf load page failed [knl_buffer_access.c:1109]
UTC+08:00 2026-08-07 14:39:29.003|CLUSTER|00000|3832741|ERROR>OG-00758:[DTC_BNUFFER][19-1][dtc buf try prefetch] failed, read num:1,The tablespace invalid space is offline, buf load page failed [dtc_buffer.c:140]
UTC+08:00 2026-08-07 14:39:29.003|BUFFER|00000|3832741|ERROR>OG-00758:[BUFFER] ABORT INFO: failed to read page 19-1,The tablespace invalid space is offline, buf load page failed [knl_buffer_access.h:94]
UTC+08:00 2026-08-07 14:39:29.005|COMMON|00000|3832741|INFO>[LOG] file '/data/ograc_install/ograc/log/ograc/blackbox/ogracd.blog' is added [cm_log.c:858]

提单组织

内部开发

测试代码

likedislike
刘凯刘凯成员
8月7日 添加了label:bug
opengauss_bot
opengauss_bot成员
8月7日 评论:

This issue requires an assignee. Since you haven't specified one, we've assigned TestManager as the default assignee for this issue.

likedislike
opengauss_botopengauss_bot成员
8月7日 将 TestManager 设为负责人
opengauss_botopengauss_bot成员
8月7日 添加了label:sig/StorageEngine
opengauss_bot
opengauss_bot成员
8月7日 评论:

Welcome To openGauss Community

Hey @Miyano , thanks for your contribution to the community.

Bot Usage Manual

I'm the Bot here serving you. You can find the instructions on how to interact with me at Here . That means you can comment below every pull request or issue to trigger Bot Commands.

Contact Guide

If you have any questions, please contact the SIG: StorageEngine ,
and any of the maintainers: @CarrotGo, @chendong76, @chenxiaobin19, @congzhou2603, @dodders, @hwworkholic, @jemappellehc, @muyulinzhong, @quemingjian, @shenzheng4, @shirley_zhengx, @superlchf, @totaj, @wlff234, @wofanzheng, @ywzq1161327784 ,
and any of the committers: @CityYard, @libiao2024, @lijieac, @theothersideofsea, @wang_bowen, @weithu, @xiong_xjun, @yuanyazhi, @zhangzq131 .

likedislike
刘凯
刘凯成员
8月7日 评论:
(gdb) f 7
#7  buf_check_resident_page_version (session=session@entry=0xfffad0f57000, page_id=...) at /home/oGRAC/pkg/src/kernel/buffer/knl_buffer.c:1828
1828    /home/oGRAC/pkg/src/kernel/buffer/knl_buffer.c: No such file or directory.
(gdb) p page_id
$1 = {value = <optimized out>, vmid = 1, {page = 1, file = <optimized out>, aligned = <optimized out>}}
(gdb) f 8
#8  0x0000aaaae22bfadc in rmon_free_spc_extents (rmon_ctx=0xffff4acf0c78, session=0xfffad0f57000) at /home/oGRAC/pkg/src/kernel/daemon/knl_rmon.c:220
220     /home/oGRAC/pkg/src/kernel/daemon/knl_rmon.c: No such file or directory.
(gdb) p i
$2 = 13
(gdb) p space->ctrl->id
$3 = 13
(gdb) p space->ctrl
$4 = (space_ctrl_t *) 0xfffacdd86038
(gdb) p *(space->ctrl)
$5 = {id = 13, used = 0, name = "\000ABSPACE_MPART_REBINDEX_NOENC_1", '\000' <repeats 36 times>, flag = 0, block_size = 8192, extent_size = 8, file_hwm = 0, 
  type = 32, org_scn = 18446744073709551615, encrypt_version = 0 '\000', cipher_reserve_size = 0 '\000', is_for_create_db = 0 '\000', 
  unused = '\000' <repeats 12 times>, files = {4294967295 <repeats 1000 times>}}
  (gdb) p space->entry
$6 = {value = 4393751543808, vmid = 0, {page = 0, file = 1023, aligned = 0}}
(gdb) p space->head
$7 = (space_head_t *) 0x0
(gdb) p space->is_empty
$8 = 0 '\000'
  
likedislike
刘凯刘凯成员
8月7日 issue优先级由 无优先级 改变为 不重要
刘凯刘凯成员
8月7日 关联了看板:openGauss 7.0.0-LTS
刘凯
刘凯成员
8月7日 评论:

1. 崩溃线程定位
core 线程是 rmon 后台线程,不是执行 DROP TABLESPACE 的前台 SQL 线程:

#7 buf_check_resident_page_version(...)
#8 rmon_free_spc_extents(...) at knl_rmon.c:220
#9 rmon_proc(...)

rmon_proc 周期性调用 rmon_free_spc_extents,用于把 bitmap tablespace 的 free_extents 链表释放回 bitmap。

2. rmon 崩溃点
崩在 knl_rmon.c:220

if (!space->ctrl->used || !SPACE_IS_ONLINE(space) || !SPACE_IS_BITMAPMANAGED(space)) {
    continue;
}

head = SPACE_HEAD_RESIDENT(session, space);

这里的问题是:前置判断和 SPACE_HEAD_RESIDENT 之间没有拿 space->lock。所以 space 状态可能在判断后被 DROP 线程改变。

3. SPACE_HEAD_RESIDENT 不是普通取指针
宏在 knl_space_base.h:141

#define SPACE_HEAD_RESIDENT(session, space) \
    ((buf_check_resident_page_version(session, (space)->entry)) ? ((space)->head) : ((space)->head))

它表面上只是返回 space->head,但实际会先调用 buf_check_resident_page_version(session, space->entry)。也就是说,这个宏有读 buffer/page 的副作用。

你的栈正是进入了 buf_check_resident_page_version

在集群分支里,如果本地 DCS page 不可直接用,会走到:

buf_enter_page(session, page_id, LATCH_MODE_S, ENTER_PAGE_NORMAL);

buf_enter_page 是 fatal wrapper:

if (buf_read_page(...) != OG_SUCCESS) {
    CM_ABORT(0, "[BUFFER] ABORT INFO: failed to read page ...");
}

所以一旦读这个 space head page 失败,就不是返回错误,而是直接 SIGABRT core。

4. DROP TABLESPACE 侧的状态变化
DROP 在线表空间路径在 knl_drop_space.c:1131space->lock,随后 line 1145

SPACE_UNSET_ONLINE(space);

再进入 spc_remove_space_online -> spc_remove_space -> spc_reset_space

spc_reset_spaceknl_drop_space.c:825 会做这些重置:

space->entry = INVALID_PAGID;
space->head = NULL;
space->ctrl->file_hwm = 0;
space->ctrl->name[0] = '\0';
space->ctrl->used = OG_FALSE;
space->ctrl->flag = 0;
files = INVALID...

5. gdb 证据和源码完全对上
你现场看到:

i = 13
space->ctrl->id = 13
used = 0
name = "\000ABSPACE_MPART_REBINDEX_NOENC_1"...
flag = 0
file_hwm = 0
files = {4294967295 <repeats 1000 times>}
space->entry = {page = 0, file = 1023}
space->head = 0x0
space->is_empty = 0

这些值就是 spc_reset_space() 之后的状态。特别是:

  • name[0] = '\0',所以还能看到残留名字从第二个字符开始;
  • files 全部 4294967295,即 OG_INVALID_ID32
  • space->entry = file 1023/page 0,基本就是 INVALID_PAGID
  • space->head = NULL
  • space->is_empty = 0 说明已经走过 spc_reset_space,不是刚设置 is_empty = TRUE 的中间态。

但 rmon 源码 line 216 明明会跳过 used=0 的 space。它还能到 line 220,唯一合理解释就是:rmon 做 line 216 判断时,space 13 还没 reset;执行到 SPACE_HEAD_RESIDENT 期间,DROP 线程完成了 reset。

6. page_id 不一致也能解释
frame 7 里:

page_id = {vmid = 1, page = 1, file = <optimized out>}

frame 8 当前:

space->entry = {page = 0, file = 1023}

这不是矛盾。SPACE_HEAD_RESIDENT(session, space) 调用时,旧的 space->entry 已经按值传给了 buf_check_resident_page_version,所以 frame 7 还拿着旧 page_id;之后 DROP reset 了 space->entry,所以 frame 8 看到的是 invalid entry。

根因
这是一个典型 TOCTOU 并发问题:

rmon_free_spc_extents 在未持有 space->lock 的情况下,先判断 space->ctrl->used/online/bitmap,随后访问带副作用的 SPACE_HEAD_RESIDENT。DROP TABLESPACE 持有同一 space->lock 重置 space,但 rmon 没有参与这把锁同步,导致 rmon 可能在表空间已 drop/reset 后继续用旧 entry 读 resident page。读页失败被 buf_enter_page 升级为 CM_ABORT,最终进程 SIGABRT。

这不是数据页损坏的证据,更像是 DROP TABLESPACE 与 rmon 后台 free extent 清理之间的并发保护缺失。

likedislike
刘凯刘凯成员
8月8日 关联了pull request:fix rmon tablespace drop race
muyulinzhongmuyulinzhong成员
8月10日 将 Miyano 设为负责人
muyulinzhongmuyulinzhong成员
8月10日 移除了负责人 TestManager
刘凯刘凯成员
8月11日 issue状态由 待办的 改变为 已完成
刘凯
刘凯成员
8月12日 评论:

B018自验

SQL> DROP USER NEBULA CASCADE;

Succeed.

SQL> CREATE USER NEBULA IDENTIFIED BY xxx;

Succeed.

SQL> GRANT ALL TO NEBULA;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

OG-00780, The tablespace xxx does not exist.
SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

OG-00780, The tablespace xxx does not exist.
SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> DROP TABLESPACE xxx INCLUDING CONTENTS AND DATAFILES;

Succeed.

SQL> quit
likedislike
刘凯刘凯成员
8月12日 issue状态由 已完成 改变为 待回归
陈博达
陈博达成员
8月18日 评论:

通过
image.png

likedislike
liuzhen001liuzhen001成员
8月19日 issue状态由 待回归 改变为 已验收
liuzhen001liuzhen001成员
8月19日 关闭了 issue