open-gauss-cm-operator:基于 Kubernetes 的 openGauss 集群管理项目

可用于在 Kubernetes 环境中部署和管理 openGauss 数据库集群,支持集群部署、主从切换、备份恢复、节点驱逐、资源扩缩容等核心功能,具备固定节点、固定IP及高可用能力,简化数据库运维。【此简介由AI生成】

Branch1Tags0

Containerization Solution Design for openGauss Based on CM Mode

1. Introduction to openGauss and CM

openGauss is an enterprise-level open-source relational database that offers high performance, high availability, strong security, and easy maintenance. Integrating CM improves the O&M and reliability of openGauss clusters. Cluster Manager (CM) is cluster resource management software. It supports customized resource monitoring and provides capabilities such as monitoring of the primary/standby database status, network communication faults, file system faults, and automatic primary/standby switchover upon faults. It also offers various cluster management capabilities, such as starting and stopping clusters, nodes, and instances, querying cluster status, performing primary node selection and primary/standby switchover, and managing logs. This solution leverages the community CM to provide high availability for openGauss while reducing the complexity of direct cluster management by the Operator.

2. Architecture Design

image-20230912110225056

  • openGaussCluster: Declares the custom resource object. When the Operator detects a create event, it will create three types of Pods:

    • openGauss: database nodes (Primary/Standby).
    • CMS: responsible for database primary election, with 3 Pods.
    • openGauss-export: collects database monitoring metrics, with 1 Pod.
  • Backup-Recovery

    • Backup: When backup is declared, a job is created. A client connects to the database service, executes a full physical backup using gs_probackup, uploads backup files to the backup server, and logs the entire process.

    • Recovery: A job is created, a client pulls backup files from the backup server, executes data recovery using gs_probackup, and logs the entire process.

  • Fluentd: Deployed via DaemonSet on openGauss hosts to collect /data disk logs.

3. Feature List

  • Cluster deployment (supports one primary with multiple standbys, fixed nodes, fixed IPs)

  • Primary/standby switchover

  • Backup and recovery

  • Standby node rebuild

  • Node eviction

  • Resource specification scaling

  • Monitoring collection

  • Log collection

  • Cluster maintenance

  • Cluster release

4. Key Capability Implementation

4.1 Fixed Nodes

​ Fixed nodes ensure that when a DN pod is rescheduled, the pod is scheduled to the same host to prevent data loss. This uses delayed scheduler and PV node affinity features.

  • Set the volume binding mode of the storageclass to WaitForFirstConsumer.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name:  local-storage-test
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
  • After pod scheduling, bind the PV to the pod. Future rescheduling references the node of the PV:
nodeAffinity:
 required:
  nodeSelectorTerms:
  - matchExpressions:
    - key: kubernetes.io/hostname
      operator: In
      values:
      - ly-xjf-r020301-gyt

4.2 Fixed IP

​ Fixed IP ensures pods retain their IPs after restart to maintain cluster topology, using Calico IPAM:

  1. Calico plugin automatically assigns pod IPs.
  2. Operator uses list-watch to write the pod IP back to the pod annotations after creation: "cni.projectcalico.org/ipAddrs":"["xx.xx.xx.xx"]"

4.3 Backup and Recovery

​ Backup and recovery capabilities are implemented via container images. The results display is as follows:

  1. Jobs implement single backup or recovery.
  2. When the job finishes, the Operator reads pod logs and determines success based on log endings, reporting results.

4.4 High Availability

CM itself provides high availability:

  • gauss process keep-alive: cm_agent checks gauss every second. If the process does not exist, it starts the process. If the process is dead, it kills the process and restarts it.
  • cm_agent keep-alive: om_monitor checks cm_agent every second. If the process does not exist, it starts the process. If the process is dead, it kills the process and restarts it.
  • om_monitor keep-alive: entrypoint.sh runs as a daemon, checks om_monitor every minute. If the process is dead, it kills the process and restarts it.

CM handles most failures, but issues remain:

  • Ensuring the entrypoint.sh daemon always exists.
  • Pods cannot auto-recover if deleted.
  • Node failures or full disks can cause cluster issues CM cannot handle.

Container-level solutions for these cases:

  • Use entrypoint.sh as the main process; if entrypoint.sh does not exist, the container will automatically restart (set the container restart policy).
  • Operator watches pod deletion events and recreates pods.
  • Node eviction handles host (of the pod) failures.
  • Add health checks:
    • Set the reconciliation synchronization frequency of the operator. The health of the resource objects monitored by the operator is checked every minute, and ogc.status is updated accordingly. If the cluster status is abnormal, warning logs are generated. A component can be used to collect warning logs and trigger cluster status exception alarms.
    • Set the readiness probe. It checks whether the GaussDB process exists. If the process does not exist, no action is taken. If the process exists, the readiness probe checks whether the service is in the normal state. If the service is not in the normal state, an exception is returned. After the readiness probe is added to the GaussDB container, it is executed every 5 seconds. If an exception is returned for more than three times, the GaussDB container status changes to Notready. In this case, the operator detects the status change, marks the cluster status as abnormal, and generates warning logs. Warning logs are collected and alarms are triggered.
    • Set the liveness probe. It checks whether the GaussDB process exists. If the process does not exist, an exception is returned. For operations performed using cm_ctl stop, the check is skipped. After the liveness probe is added to the GaussDB container, it is executed every 5 seconds. If an exception is returned for more than three times, the GaussDB container is restarted.

5. Installation

5.1 Image Building

5.1.1 Operator Image

Download the operator code.

git clone -b dev-20230620 https://gitee.com/youshouldnowme/openGauss-operator

Generate an EXE file.

set GOARCH=amd64
set GOOS=linux
set GO111MODULE=off
SET CGO_ENABLED=0
go build -a -o opengauss_operator cmd/main.go

Dockerfile

FROM k8s-deploy/busybox:latest
ADD opengauss_operator /
ADD customConfig /customConfig
RUN chmod 111 opengauss_operator

Create an operator image based on the Dockerfile.

docker build -t k8s-deploy/opengauss-operator:v5.0.0-20230930

5.1.2 openGauss Image

git clone -b dev-multidn https://gitee.com/wang-xinyu-7/openGauss-image

5.1.3 Backup-Recovery Image

git clone -b backup-restore https://gitee.com/wang-xinyu-7/openGauss-image

5.1.4 openGauss-exporter Image

git clone https://gitee.com/opengauss/openGauss-prometheus-exporter

5.1.5 Fluented Image

git clone https://github.com/fluent/fluentd

5.2 Deployment

5.2.1 Deploying CRD

Deploy ogc:

kubectl apply -f config\crd\bases\gaussdb.gaussdb.middleware.cmos.cn_opengaussbackuprecoveries.yaml

Deploy ogbr:

kubectl apply -f config\crd\bases\gaussdb.gaussdb.middleware.cmos.cn_opengaussclusters.yaml

5.2.2 Deploying the Operator

Deploy the operator deployment:

kubectl apply -f deploy/operator-deployment.yaml

5.2.3 Deploying an openGauss Cluster

Deploy:

kubectl apply -f config\samples\gaussdb_v1_opengausscluster.yaml

Check the cluster:

kubectl get opengaussclusters.gaussdb.gaussdb.middleware.cmos.cn

6. User Guide

6.1 Custom Resource Definitions (CRDs)

6.1.1 openGaussCluster

Property Type Description
Architecture string Deployment architecture. split = CM and DN separated, merge = CM+DN integrated (not supported)
NetworkType string Network type: overlay/underlay
DataNodeNum *int32 Number of DN nodes
Maintenance bool true = manual maintenance, false = Operator-managed
Images ContainerImage Image set (DN, backup/recovery, monitoring)
DBResources ResourceList DN container resource specifications
DBStorage DBStorageTemplate Storage template
Ports *int32 External service port. In overlay mode, it refers to the start port.
BackRecovery BackRecoverySpec Backup/recovery info
Switchover SwitchoverMessage Primary/standby switchover info
StandbyReconstruct StandbyReconstructSpec Standby node rebuild info
IsInit bool After the cluster status changes to Running for the first time, initialization is performed. The initialization includes creating the GaussDB account and password and setting permissions.
InstanceMigration InstanceMigrationSpec Node eviction info

6.1.2 openGaussBackupRecovery

Property Type Description
Address string Backup server address
Mode string Backup file transfer mode (only scp supported)
Time string Scheduled backup time
BackupFile string Backup file name

6.2 Cluster Deployment

Deploy:

kubectl apply -f zhengxin-test.yaml

Check the cluster status. If the state is Running, the cluster is normal. You can also view information about each node and the primary/secondary relationship.

kubectl get ogc zhengxin-test -oyaml

image-20230911152702794

6.3 External Access to Services

Check NodePort:

kubectl get service -n cmdb

Find pod host IP:

kubectl get pod -n cmdb -owide  

kubectl describe node ${nodeName} | grep IP

Connect to the database service via a client.

image-20230911152902359

6.4 Primary/Standby Switchover

Check current primary node:

kubectl get ogc zhengxin-test -oyaml

Declare a primary-standby switch:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  # Primary/standby switchover
  switchover:
    targetMasterPod: zhengxin-test-0
    status: begin

Check switch result:

kubectl get ogc -n cmdb zhengxin-test -oyaml | grep -A 3 switchover

Log in to the console and check whether the primary/standby switchover is successful:

kubectl exec -it -n cmdb zhengxin-test-0 bash

su - omm
cm_ctl query -Cv

6.5 Single Backup

Declare a single backup for a node:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  backRecovery:
    transfer:
      address: 192.168.29.104
      mode: scp
    backups:
    - podName: zhengxin-test-0
      status: Create

Check backup details:

kubectl get ogc -n cmdb  zhengxin-test  -oyaml | grep -A 10 backRecovery

Check pod backup logs:

kubectl logs -f -n cmdb ${podName}

After the backup is complete, log in to the backup server 192.168.29.104 and check the backup file:

ll /data/backups/${fileName}

6.6 Scheduled Backup

Declare scheduled backup:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  backRecovery:
    cronBackupTime: 0 1 * * *
    transfer:
      address: 192.168.29.104
      mode: scp

Check cronjob:

kubectl get cronjob -n cmdb | grep zhengxin

Check scheduled backup pod after it triggers:

kubectl get pod -n cmdb | grep zhengxin | grep cron

6.7 Single Recovery

Declare a single recovery:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  backRecovery:
    transfer:
      address: 192.168.29.104
      mode: scp
    recoveries:
    - podName: ${podName}
      backFileName: opengaussbk_xxx/RYLAxx
      status: Create

Check recovery details:

kubectl get ogc -n cmdb  zhengxin-test  -oyaml | grep -A 10 backRecovery

Check recovery pod logs:

kubectl logs -f -n cmdb ${podName}

Check the cluster status:

kubectl get ogc -n cmdb zhengxin-test -oyaml

6.8 Standby Node Rebuild

Declare standby node rebuild:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: opengausscluster-sample
  namespace: ${namespace}
spec:
.......
  standbyReconstruct:
    executeInstance: ${podName}
    problemInstance: ${podName}
    status: create

Check standby node rebuild details.

kubectl get ogc -n cmdb  zhengxin-test -oyaml | grep -A 4 standbyReconstruct

When the value of standbyReconstruct.status changes to succeed, check the cluster status:

kubectl get ogc -n cmdb  zhengxin-test -oyaml 

6.9 Node Eviction

Declare node eviction:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  instanceMigration:
    instance: ${podName}
    status: create

Check node eviction details:

kubectl get ogc -n cmdb  zhengxin-test -oyaml | grep -A 3 instanceMigration

When the value of instanceMigration.status changes to succeed, check the cluster status. If the pod status is normal but the cluster status is abnormal, rebuild the standby node or perform a single recovery.

kubectl get ogc -n cmdb zhengxin-test -oyaml

6.10 Resource Specification Scaling

Declare resource specification scaling:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  dbResources:
    limits:
      cpu: 5
      memory: 15Gi
    requests:
      cpu: 2
      memory: 2Gi

Check the pod specifications:

kubectl get pod -n cmdb ${podName} -oyaml | grep -A 6 resources

Check the cluster status:

kubectl get ogc -n cmdb zhengxin-test -oyaml

6.11 Monitoring Collection

After the cluster status becomes normal, the system starts to collect database monitoring metrics. Get the IP of the monitoring pod:

kubectl describe pod -n cmdb  zhengxin-test-exporter |  grep IP

Check collected metrics:

curl ${ip}:9187/metrics

6.12 Log Collection

Find the log collection pod:

kubectl get pod -n kube-system | grep fluentd

Check the log collection configuration:

kubectl get cm -n kube-system | grep fluentd

View logs from the log collection pod:

kubectl logs -f -n kube-system ${podName}

6.13 Cluster Maintenance

Declare cluster maintenance:

apiVersion: gaussdb.gaussdb.middleware.cmos.cn/v1
kind: OpenGaussCluster
metadata:
  name: zhengxin-test
  namespace: ${namespace}
spec:
.......
  maintenance: true

During maintenance, operations like primary-standby switch or single backup will not be executed.

6.14 Cluster Release

Release the cluster:

kubectl delete -f zhengxin-test.yaml

Check the resource object. It has been deleted.

kubectl get ogc -n cmdb | grep zhengxin-test

kubectl get service -n cmdb  | grep zhengxin-test

kubectl get pvc -n cmdb  | grep zhengxin-test

kubectl get pod -n cmdb | grep zhengxin-test

Check the persistent data. It has been cleared.

ll /data/cmdb/zhengxin-test/

Introduction

可用于在 Kubernetes 环境中部署和管理 openGauss 数据库集群,支持集群部署、主从切换、备份恢复、节点驱逐、资源扩缩容等核心功能,具备固定节点、固定IP及高可用能力,简化数据库运维。【此简介由AI生成】

Customize your domain