可用于在 Kubernetes 环境中部署和管理 openGauss 数据库集群,支持集群部署、主从切换、备份恢复、节点驱逐、资源扩缩容等核心功能,具备固定节点、固定IP及高可用能力,简化数据库运维。【此简介由AI生成】
Access rate limit exceeded. Please log in to continue.
Containerization Solution Design for openGauss Based on CM Mode
1. Introduction to openGauss and CM
openGauss is an enterprise-level open-source relational database that offers high performance, high availability, strong security, and easy maintenance. Integrating CM improves the O&M and reliability of openGauss clusters. Cluster Manager (CM) is cluster resource management software. It supports customized resource monitoring and provides capabilities such as monitoring of the primary/standby database status, network communication faults, file system faults, and automatic primary/standby switchover upon faults. It also offers various cluster management capabilities, such as starting and stopping clusters, nodes, and instances, querying cluster status, performing primary node selection and primary/standby switchover, and managing logs. This solution leverages the community CM to provide high availability for openGauss while reducing the complexity of direct cluster management by the Operator.
2. Architecture Design

-
openGaussCluster: Declares the custom resource object. When the Operator detects a create event, it will create three types of Pods:
- openGauss: database nodes (Primary/Standby).
- CMS: responsible for database primary election, with 3 Pods.
- openGauss-export: collects database monitoring metrics, with 1 Pod.
-
Backup-Recovery
-
Backup: When backup is declared, a job is created. A client connects to the database service, executes a full physical backup using
gs_probackup, uploads backup files to the backup server, and logs the entire process. -
Recovery: A job is created, a client pulls backup files from the backup server, executes data recovery using
gs_probackup, and logs the entire process.
-
-
Fluentd: Deployed via DaemonSet on openGauss hosts to collect
/datadisk logs.
3. Feature List
-
Cluster deployment (supports one primary with multiple standbys, fixed nodes, fixed IPs)
-
Primary/standby switchover
-
Backup and recovery
-
Standby node rebuild
-
Node eviction
-
Resource specification scaling
-
Monitoring collection
-
Log collection
-
Cluster maintenance
-
Cluster release
4. Key Capability Implementation
4.1 Fixed Nodes
Fixed nodes ensure that when a DN pod is rescheduled, the pod is scheduled to the same host to prevent data loss. This uses delayed scheduler and PV node affinity features.
- Set the volume binding mode of the
storageclasstoWaitForFirstConsumer.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage-test
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
- After pod scheduling, bind the PV to the pod. Future rescheduling references the node of the PV:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- ly-xjf-r020301-gyt
4.2 Fixed IP
Fixed IP ensures pods retain their IPs after restart to maintain cluster topology, using Calico IPAM:
- Calico plugin automatically assigns pod IPs.
- Operator uses
list-watchto write the pod IP back to the pod annotations after creation: "cni.projectcalico.org/ipAddrs":"["xx.xx.xx.xx"]"
4.3 Backup and Recovery
Backup and recovery capabilities are implemented via container images. The results display is as follows:
- Jobs implement single backup or recovery.
- When the job finishes, the Operator reads pod logs and determines success based on log endings, reporting results.
4.4 High Availability
CM itself provides high availability:
- gauss process keep-alive:
cm_agentchecks gauss every second. If the process does not exist, it starts the process. If the process is dead, it kills the process and restarts it. - cm_agent keep-alive:
om_monitorcheckscm_agentevery second. If the process does not exist, it starts the process. If the process is dead, it kills the process and restarts it. - om_monitor keep-alive:
entrypoint.shruns as a daemon, checksom_monitorevery minute. If the process is dead, it kills the process and restarts it.
CM handles most failures, but issues remain:
- Ensuring the
entrypoint.shdaemon always exists. - Pods cannot auto-recover if deleted.
- Node failures or full disks can cause cluster issues CM cannot handle.
Container-level solutions for these cases:
- Use
entrypoint.shas the main process; ifentrypoint.shdoes not exist, the container will automatically restart (set the container restart policy). - Operator watches pod deletion events and recreates pods.
- Node eviction handles host (of the pod) failures.
- Add health checks:
- Set the reconciliation synchronization frequency of the operator. The health of the resource objects monitored by the operator is checked every minute, and
ogc.statusis updated accordingly. If the cluster status is abnormal, warning logs are generated. A component can be used to collect warning logs and trigger cluster status exception alarms. - Set the readiness probe. It checks whether the GaussDB process exists. If the process does not exist, no action is taken. If the process exists, the readiness probe checks whether the service is in the normal state. If the service is not in the normal state, an exception is returned. After the readiness probe is added to the GaussDB container, it is executed every 5 seconds. If an exception is returned for more than three times, the GaussDB container status changes to Notready. In this case, the operator detects the status change, marks the cluster status as abnormal, and generates warning logs. Warning logs are collected and alarms are triggered.
- Set the liveness probe. It checks whether the GaussDB process exists. If the process does not exist, an exception is returned. For operations performed using
cm_ctl stop, the check is skipped. After the liveness probe is added to the GaussDB container, it is executed every 5 seconds. If an exception is returned for more than three times, the GaussDB container is restarted.
- Set the reconciliation synchronization frequency of the operator. The health of the resource objects monitored by the operator is checked every minute, and
5. Installation
5.1 Image Building
5.1.1 Operator Image
Download the operator code.
git clone -b dev-20230620 https://gitee.com/youshouldnowme/openGauss-operator
Generate an EXE file.
set GOARCH=amd64
set GOOS=linux
set GO111MODULE=off
SET CGO_ENABLED=0
go build -a -o opengauss_operator cmd/main.go
Dockerfile
FROM k8s-deploy/busybox:latest
ADD opengauss_operator /
ADD customConfig /customConfig
RUN chmod 111 opengauss_operator
Create an operator image based on the Dockerfile.
docker build -t k8s-deploy/opengauss-operator:v5.0.0-20230930
5.1.2 openGauss Image
git clone -b dev-multidn https://gitee.com/wang-xinyu-7/openGauss-image
5.1.3 Backup-Recovery Image
git clone -b backup-restore https://gitee.com/wang-xinyu-7/openGauss-image
5.1.4 openGauss-exporter Image
git clone https://gitee.com/opengauss/openGauss-prometheus-exporter
5.1.5 Fluented Image
git clone https://github.com/fluent/fluentd
5.2 Deployment
5.2.1 Deploying CRD
Deploy ogc:
kubectl apply -f config\crd\bases\gaussdb.gaussdb.middleware.cmos.cn_opengaussbackuprecoveries.yaml
Deploy ogbr:
kubectl apply -f config\crd\bases\gaussdb.gaussdb.middleware.cmos.cn_opengaussclusters.yaml
5.2.2 Deploying the Operator
Deploy the operator deployment:
kubectl apply -f deploy/operator-deployment.yaml
5.2.3 Deploying an openGauss Cluster
Deploy:
kubectl apply -f config\samples\gaussdb_v1_opengausscluster.yaml
Check the cluster:
kubectl get opengaussclusters.gaussdb.gaussdb.middleware.cmos.cn
6. User Guide
6.1 Custom Resource Definitions (CRDs)
6.1.1 openGaussCluster
| Property | Type | Description |
|---|---|---|
| Architecture | string | Deployment architecture. split = CM and DN separated, merge = CM+DN integrated (not supported) |
| NetworkType | string | Network type: overlay/underlay |
| DataNodeNum | *int32 | Number of DN nodes |
| Maintenance | bool | true = manual maintenance, false = Operator-managed |
| Images | ContainerImage | Image set (DN, backup/recovery, monitoring) |
| DBResources | ResourceList | DN container resource specifications |
| DBStorage | DBStorageTemplate | Storage template |
| Ports | *int32 | External service port. In overlay mode, it refers to the start port. |
| BackRecovery | BackRecoverySpec | Backup/recovery info |
| Switchover | SwitchoverMessage | Primary/standby switchover info |
| StandbyReconstruct | StandbyReconstructSpec | Standby node rebuild info |
| IsInit | bool | After the cluster status changes to Running for the first time, initialization is performed. The initialization includes creating the GaussDB account and password and setting permissions. |
| InstanceMigration | InstanceMigrationSpec | Node eviction info |
6.1.2 openGaussBackupRecovery
| Property | Type | Description |
|---|---|---|
| Address | string | Backup server address |
| Mode | string | Backup file transfer mode (only scp supported) |
| Time | string | Scheduled backup time |
| BackupFile | string | Backup file name |
6.2 Cluster Deployment
Deploy:
kubectl apply -f zhengxin-test.yaml
Check the cluster status. If the state is Running, the cluster is normal. You can also view information about each node and the primary/secondary relationship.
kubectl get ogc zhengxin-test -oyaml

6.3 External Access to Services
Check NodePort:
kubectl get service -n cmdb
Find pod host IP:
kubectl get pod -n cmdb -owide
kubectl describe node ${nodeName} | grep IP
Connect to the database service via a client.

6.4 Primary/Standby Switchover
Check current primary node:
kubectl get ogc zhengxin-test -oyaml
Declare a primary-standby switch:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
# Primary/standby switchover
switchover:
targetMasterPod: zhengxin-test-0
status: begin
Check switch result:
kubectl get ogc -n cmdb zhengxin-test -oyaml | grep -A 3 switchover
Log in to the console and check whether the primary/standby switchover is successful:
kubectl exec -it -n cmdb zhengxin-test-0 bash
su - omm
cm_ctl query -Cv
6.5 Single Backup
Declare a single backup for a node:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
backRecovery:
transfer:
address: 192.168.29.104
mode: scp
backups:
- podName: zhengxin-test-0
status: Create
Check backup details:
kubectl get ogc -n cmdb zhengxin-test -oyaml | grep -A 10 backRecovery
Check pod backup logs:
kubectl logs -f -n cmdb ${podName}
After the backup is complete, log in to the backup server 192.168.29.104 and check the backup file:
ll /data/backups/${fileName}
6.6 Scheduled Backup
Declare scheduled backup:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
backRecovery:
cronBackupTime: 0 1 * * *
transfer:
address: 192.168.29.104
mode: scp
Check cronjob:
kubectl get cronjob -n cmdb | grep zhengxin
Check scheduled backup pod after it triggers:
kubectl get pod -n cmdb | grep zhengxin | grep cron
6.7 Single Recovery
Declare a single recovery:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
backRecovery:
transfer:
address: 192.168.29.104
mode: scp
recoveries:
- podName: ${podName}
backFileName: opengaussbk_xxx/RYLAxx
status: Create
Check recovery details:
kubectl get ogc -n cmdb zhengxin-test -oyaml | grep -A 10 backRecovery
Check recovery pod logs:
kubectl logs -f -n cmdb ${podName}
Check the cluster status:
kubectl get ogc -n cmdb zhengxin-test -oyaml
6.8 Standby Node Rebuild
Declare standby node rebuild:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: opengausscluster-sample
namespace: ${namespace}
spec:
.......
standbyReconstruct:
executeInstance: ${podName}
problemInstance: ${podName}
status: create
Check standby node rebuild details.
kubectl get ogc -n cmdb zhengxin-test -oyaml | grep -A 4 standbyReconstruct
When the value of standbyReconstruct.status changes to succeed, check the cluster status:
kubectl get ogc -n cmdb zhengxin-test -oyaml
6.9 Node Eviction
Declare node eviction:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
instanceMigration:
instance: ${podName}
status: create
Check node eviction details:
kubectl get ogc -n cmdb zhengxin-test -oyaml | grep -A 3 instanceMigration
When the value of instanceMigration.status changes to succeed, check the cluster status. If the pod status is normal but the cluster status is abnormal, rebuild the standby node or perform a single recovery.
kubectl get ogc -n cmdb zhengxin-test -oyaml
6.10 Resource Specification Scaling
Declare resource specification scaling:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
dbResources:
limits:
cpu: 5
memory: 15Gi
requests:
cpu: 2
memory: 2Gi
Check the pod specifications:
kubectl get pod -n cmdb ${podName} -oyaml | grep -A 6 resources
Check the cluster status:
kubectl get ogc -n cmdb zhengxin-test -oyaml
6.11 Monitoring Collection
After the cluster status becomes normal, the system starts to collect database monitoring metrics. Get the IP of the monitoring pod:
kubectl describe pod -n cmdb zhengxin-test-exporter | grep IP
Check collected metrics:
curl ${ip}:9187/metrics
6.12 Log Collection
Find the log collection pod:
kubectl get pod -n kube-system | grep fluentd
Check the log collection configuration:
kubectl get cm -n kube-system | grep fluentd
View logs from the log collection pod:
kubectl logs -f -n kube-system ${podName}
6.13 Cluster Maintenance
Declare cluster maintenance:
apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
name: zhengxin-test
namespace: ${namespace}
spec:
.......
maintenance: true
During maintenance, operations like primary-standby switch or single backup will not be executed.
6.14 Cluster Release
Release the cluster:
kubectl delete -f zhengxin-test.yaml
Check the resource object. It has been deleted.
kubectl get ogc -n cmdb | grep zhengxin-test
kubectl get service -n cmdb | grep zhengxin-test
kubectl get pvc -n cmdb | grep zhengxin-test
kubectl get pod -n cmdb | grep zhengxin-test
Check the persistent data. It has been cleared.
ll /data/cmdb/zhengxin-test/
Introduction
可用于在 Kubernetes 环境中部署和管理 openGauss 数据库集群,支持集群部署、主从切换、备份恢复、节点驱逐、资源扩缩容等核心功能,具备固定节点、固定IP及高可用能力,简化数据库运维。【此简介由AI生成】
Customize your domain