Star4
32
代码
代码
Issues19
Pull Requests9
项目讨论
Wiki
分析
Star4
32
已开启
docs(design): add BKE design documentation #211
zhangzq1011创建于 8月4日
docs(design): add BKE design documentation #211
已开启
zhangzq1011创建于 8月4日
从zhangzq1011/sig-installationdocs/add-bke-design合入到openFuyao/sig-installationmaster
共 47 个文件变更+3696-0
Adesign/BKE设计文档/README.md+57-0
@@ -0,0 +1,57 @@
1+# BKE 设计文档
2+ 
3+BKE的目标是实现k8s集群以及内部自研组件在裸机中标准化部署，在该仓库中主要涉及bke产品的全局设计以及部分技术相关的调研工作
4+ 
5+#### 概览
6+ 
7+##### [诞生背景](概览/bkeadm.pdf)
8+ 
9+##### [架构](概览/架构.md)
10+ 
11+##### [项目简要](概览/项目简要.md)
12+ 
13+##### [环境要求](概览/环境要求.md)
14+ 
15+##### [使用手册](概览/使用手册.md)
16+ 
17+#### 详细设计
18+ 
19+##### [研发要求](详细设计/研发要求.md)
20+ 
21+##### [bke目录](详细设计/bke目录.md)
22+ 
23+##### [bkeadm](详细设计/bkeadm.md)
24+ 
25+##### [bkeagent](详细设计/bkeagent.md)
26+ 
27+##### [cluster-api-provider-metal](详细设计/cluster-api-provider-metal.md)
28+ 
29+##### [manifests](详细设计/manifests.md)
30+ 
31+##### [cluster-api-provider-controllerplane](详细设计/cluster-api-provider-controllerplane.md)
32+ 
33+#### 相关技术
34+ 
35+> - 部分关键技术文章
36+ 
37+#### cluster-api
38+ 
39+##### [cluster-api功能介绍](cluster-api/cluster-api功能介绍.md)
40+ 
41+##### [k8s证书续期管理机制](cluster-api/k8s证书续期管理机制.md)
42+ 
43+##### [如何伸缩k8s节点](cluster-api/如何伸缩k8s节点.md)
44+ 
45+##### [kube-apiserver可选参数配置](cluster-api/kube-apiserver可选参数配置.md)
46+ 
47+##### [kube-controller可选参数配置](cluster-api/kube-controller可选参数配置.md)
48+ 
49+##### [kube-scheduler可选参数配置](cluster-api/kube-scheduler可选参数配置.md)
50+ 
51+##### [kubeadm可选参数配置](cluster-api/kubeadm可选参数配置.md)
52+ 
53+##### [kubelet可选参数配置](cluster-api/kubelet可选参数配置.md)
54+ 
55+##### [单主多节点](cluster-api/单主多节点.yaml)
56+ 
57+##### [多主多节点](cluster-api/多主多节点.yaml)
Adesign/BKE设计文档/cluster-api/cluster-api功能介绍.md+885-0
@@ -0,0 +1,885 @@
1+#### 简介
2+ 
3+Cluster API 是一个 Kubernetes 子项目，专注于提供声明性 API 和工具来简化配置、升级和操作多个 Kubernetes 集群。
4+ 
5+通俗易懂的来说是通过创建yml来配置一个k8s集群，然后由Cluster API为用户搭建一套k8s集群。
6+ 
7+#### 架构
8+ 
9+![](https://cluster-api.sigs.k8s.io/images/management-cluster.svg)
10+ 
11+#### 各个组件介绍
12+ 
13+Cluster API总共拥有4个组件，各个组件之间相互配合共同完成集群的部署
14+ 
15+- ##### Core Provider（核心提供者）
16+ 
17+ - cluster-api 是 Cluster API 的核心组件，所有的组件的行动都的听从CoreProvider的指挥
18+ - 不可替换
19+ 
20+ - 编译入口
21+ 
22+ ```bash
23+ cd /path/to/sigs.k8s.io/cluster-api
24+ LDFLAGS=$(shell hack/version.sh)
25+ BIN_DIR=bin
26+ go build -trimpath -ldflags "$(LDFLAGS)" -o $(BIN_DIR)/manager sigs.k8s.io/cluster-api
27+ ```
28+ 
29+ 或者
30+ 
31+ ```bash
32+ cd /path/to/sigs.k8s.io/cluster-api
33+ make manager-core
34+ ```
35+ 
36+- ##### Bootstrap Provider（引导程序提供者）
37+ 
38+ - kubeadm-bootstrap-provider是Cluster API的默认Bootstrap Provider
39+ - 负责kubeadm init||join 时的 kubeadm.config的生成，并将数据写入到Secret中
40+ - 可选择其他Bootstrap Provider进行替换
41+ 
42+ - 编译入口
43+ 
44+ ```bash
45+ cd /path/to/sigs.k8s.io/cluster-api
46+ LDFLAGS=$(shell hack/version.sh)
47+ BIN_DIR=bin
48+ go build -trimpath -ldflags "$(LDFLAGS)" -o $(BIN_DIR)/kubeadm-bootstrap-manager sigs.k8s.io/cluster-api/bootstrap/kubeadm
49+ ```
50+ 
51+ 或者
52+ 
53+ ```bash
54+ cd /path/to/sigs.k8s.io/cluster-api
55+ make manager-kubeadm-bootstrap
56+ ```
57+ 
58+- ##### Control Plane Provider（控制平面提供者）
59+ 
60+ - kubeadm-control-plane-provider是Cluster API 的默认Control Plane Provider
61+ - 负责Target Cluster控制平面的节点生成，以及kubeconfig的生成。
62+ - 可选择其他ControlPlaneProvider进行替换
63+ 
64+ - 编译入口
65+ 
66+ ```bash
67+ cd /path/to/sigs.k8s.io/cluster-api
68+ LDFLAGS=$(shell hack/version.sh)
69+ BIN_DIR=bin
70+ go build -trimpath -ldflags "$(LDFLAGS)" -o $(BIN_DIR)/kubeadm-control-plane-manager sigs.k8s.io/cluster-api/controlplane/kubeadm
71+ ```
72+ 
73+ 或者
74+ 
75+ ```bash
76+ cd /path/to/sigs.k8s.io/cluster-api
77+ make manager-kubeadm-control-plane
78+ ```
79+ 
80+- ##### Infrastructure Provider（基础设施提供者）
81+ 
82+ - Cluster API 没有默认的Infrastructure Provider，不过许多组织和云厂商都推出了自己的Infrastructure Provider
83+ 
84+ - 负责提供实际的计算资源，如虚机或者裸机
85+ 
86+ - 例如，云基础设施提供商包括 AWS、Azure 和 Google，而裸机基础设施提供商包括 VMware、MAAS 和 metal3.io
87+ 
88+ - 编译入口
89+ 
90+ 略，根据实际使用的Infrastructure Provider决定
91+ 
92+#### CRD介绍
93+ 
94+以Cluster API的默认Bootstrap Provider（kubeadm-bootstrap-provider）和Control Plane Provider（kubeadm-control-plane-provider），以及额外的Infrastructure Provider（cluster-api-provider-docker）为例，并以以上配置一个单master单worker的Target Cluster
95+ 
96+- ##### 先来一份全家福（https://github.com/kubernetes-sigs/cluster-api/blob/main/test/infrastructure/docker/examples/simple-cluster.yaml）
97+ 
98+ 另外需注意，定义一个集群时需要将所有CRD放置在同一Namespace中，如下面实例的Namespace为default，建议在实际使用时，不同集群使用不同的Namespace加以区分
99+ 
100+ ```yaml
101+ # Creates a cluster with one control-plane node and one worker node
102+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
103+ kind: DockerCluster
104+ metadata:
105+ name: my-cluster
106+ namespace: default
107+ ---
108+ apiVersion: cluster.x-k8s.io/v1beta1
109+ kind: Cluster
110+ metadata:
111+ name: my-cluster
112+ namespace: default
113+ spec:
114+ clusterNetwork:
115+ services:
116+ cidrBlocks: ["10.96.0.0/12"]
117+ pods:
118+ cidrBlocks: ["192.168.0.0/16"]
119+ serviceDomain: "cluster.local"
120+ controlPlaneRef:
121+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
122+ kind: KubeadmControlPlane
123+ name: controlplane
124+ namespace: default
125+ infrastructureRef:
126+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
127+ kind: DockerCluster
128+ name: my-cluster
129+ namespace: default
130+ ---
131+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
132+ kind: DockerMachineTemplate
133+ metadata:
134+ name: controlplane
135+ namespace: default
136+ spec:
137+ template:
138+ spec: {}
139+ ---
140+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
141+ kind: KubeadmControlPlane
142+ metadata:
143+ name: controlplane
144+ namespace: default
145+ spec:
146+ replicas: 1
147+ version: v1.23.3
148+ machineTemplate:
149+ infrastructureRef:
150+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
151+ kind: DockerMachineTemplate
152+ name: controlplane
153+ namespace: default
154+ kubeadmConfigSpec:
155+ clusterConfiguration:
156+ apiServer:
157+ certSANs:
158+ - localhost
159+ - 127.0.0.1
160+ - 0.0.0.0
161+ controllerManager:
162+ extraArgs:
163+ enable-hostpath-provisioner: "true"
164+ initConfiguration:
165+ nodeRegistration:
166+ kubeletExtraArgs:
167+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
168+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
169+ cgroup-driver: cgroupfs
170+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
171+ joinConfiguration:
172+ nodeRegistration:
173+ kubeletExtraArgs:
174+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
175+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
176+ cgroup-driver: cgroupfs
177+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
178+ ---
179+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
180+ kind: DockerMachineTemplate
181+ metadata:
182+ name: worker
183+ namespace: default
184+ spec:
185+ template:
186+ spec: {}
187+ ---
188+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
189+ kind: KubeadmConfigTemplate
190+ metadata:
191+ name: worker
192+ spec:
193+ template:
194+ spec:
195+ joinConfiguration:
196+ nodeRegistration:
197+ kubeletExtraArgs:
198+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
199+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
200+ cgroup-driver: cgroupfs
201+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
202+ ---
203+ apiVersion: cluster.x-k8s.io/v1beta1
204+ kind: MachineDeployment
205+ metadata:
206+ name: worker-md-0
207+ spec:
208+ clusterName: my-cluster
209+ replicas: 1
210+ selector:
211+ matchLabels:
212+ cluster.x-k8s.io/cluster-name: my-cluster
213+ template:
214+ spec:
215+ version: v1.23.3
216+ clusterName: my-cluster
217+ bootstrap:
218+ configRef:
219+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
220+ kind: KubeadmConfigTemplate
221+ name: worker
222+ infrastructureRef:
223+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
224+ kind: DockerMachineTemplate
225+ name: worker
226+ ```
227+ 
228+- ##### cluster-api （Core Provider）
229+ 
230+ 拥有6个CRD，但经常使用的只有5种，必须要在yaml中配置`Cluster`、`MachineDeployment`（没有列出例子的代表为可选配置）
231+ 
232+ - Cluster
233+ 
234+ ```yaml
235+ apiVersion: cluster.x-k8s.io/v1beta1
236+ kind: Cluster
237+ metadata:
238+ name: my-cluster
239+ namespace: default
240+ spec:
241+ # 集群网络
242+ clusterNetwork:
243+ services:
244+ cidrBlocks: ["10.96.0.0/12"]
245+ pods:
246+ cidrBlocks: ["192.168.0.0/16"]
247+ serviceDomain: "cluster.local"
248+ # 依赖的控制平面提供者资源
249+ controlPlaneRef:
250+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
251+ kind: KubeadmControlPlane
252+ name: controlplane
253+ namespace: default
254+ # 依赖的基础设施提供者资源
255+ infrastructureRef:
256+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
257+ kind: DockerCluster
258+ name: my-cluster
259+ namespace: default
260+ ```
261+ 
262+ - MachineDeployment
263+ 
264+ 主要是对Target Cluster worker节点的配置
265+ 
266+ ```yaml
267+ apiVersion: cluster.x-k8s.io/v1beta1
268+ kind: MachineDeployment
269+ metadata:
270+ name: worker-md-0
271+ spec:
272+ clusterName: my-cluster
273+ # worker节点数
274+ replicas: 1
275+ selector:
276+ matchLabels:
277+ cluster.x-k8s.io/cluster-name: my-cluster
278+ template:
279+ spec:
280+ # worker节点k8s版本
281+ version: v1.23.3
282+ clusterName: my-cluster
283+ # 依赖的引导程序提供者资源
284+ bootstrap:
285+ configRef:
286+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
287+ kind: KubeadmConfigTemplate
288+ name: worker
289+ # 依赖的基础设施提供者资源
290+ infrastructureRef:
291+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
292+ kind: DockerMachineTemplate
293+ name: worker
294+ # 机器替换升级策略
295+ # https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html#upgrading-machines-managed-by-a-machinedeployment
296+ # strategy:
297+ # type: RollingUpdate || OnDelete
298+ # type = RollingUpdate 启用
299+ # rollingUpdate:
300+ ```
301+ 
302+ - MachineSet
303+ 
304+ 由cluster-api 核心组件自动填充字段
305+ 
306+ - Machine
307+ 
308+ 由cluster-api 核心组件自动填充字段
309+ 
310+ - Machinehealthcheck
311+ 
312+ 略，可选配置，详细配置可参考 [link](https://cluster-api.sigs.k8s.io/tasks/healthcheck.html)
313+ 
314+ - ClusterClass
315+ 
316+ 目前处于alpha状态，目的是减少yaml文件并支持灵活而强大的集群定制
317+ 
318+- ##### kubeadm-bootstrap-provider（Bootstrap Provider）
319+ 
320+ KubeadmConfigTemplate必须在yaml中配置，并在MachineDeployment关联
321+ 
322+ - KubeadmConfigTemplate
323+ 
324+ 是在worker节点kubeadm join的配置，详细配置可参考kubeadm config配置
325+ 
326+ ```yaml
327+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
328+ kind: KubeadmConfigTemplate
329+ metadata:
330+ name: worker
331+ spec:
332+ template:
333+ spec:
334+ joinConfiguration:
335+ nodeRegistration:
336+ kubeletExtraArgs:
337+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
338+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
339+ cgroup-driver: cgroupfs
340+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
341+ ```
342+ 
343+ - KubeadmConfig
344+ 
345+ 由KubeadmConfigTemplate自动填充字段
346+ 
347+- ##### kubeadm-control-plane-provider（Control Plane Provider）
348+ 
349+ KubeadmControlPlane必须在yaml中配置，并在Cluster中关联
350+ 
351+ - KubeadmControlPlane
352+ 
353+ ```yaml
354+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
355+ kind: KubeadmControlPlane
356+ metadata:
357+ name: controlplane
358+ namespace: default
359+ spec:
360+ # master节点个数
361+ replicas: 1
362+ # master节点k8s版本
363+ version: v1.23.3
364+ # 机器升级替换策略，现在只有一种 RollingUpdate
365+ # RolloutStrategy:
366+ # type: RollingUpdate
367+ # type=RollingUpdate时启用
368+ # rollingUpdate:
369+ # maxSurge: 1
370+ machineTemplate:
371+ # 依赖的基础设施提供者资源
372+ infrastructureRef:
373+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
374+ kind: DockerMachineTemplate
375+ name: controlplane
376+ namespace: default
377+ # master节点kubeadm init || join 配置，详细配置可参考kubeadm config配置
378+ kubeadmConfigSpec:
379+ clusterConfiguration:
380+ apiServer:
381+ certSANs:
382+ - localhost
383+ - 127.0.0.1
384+ - 0.0.0.0
385+ controllerManager:
386+ extraArgs:
387+ enable-hostpath-provisioner: "true"
388+ initConfiguration:
389+ nodeRegistration:
390+ kubeletExtraArgs:
391+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
392+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
393+ cgroup-driver: cgroupfs
394+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
395+ joinConfiguration:
396+ nodeRegistration:
397+ kubeletExtraArgs:
398+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
399+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
400+ cgroup-driver: cgroupfs
401+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
402+ ```
403+ 
404+ - KubeadmControlPlaneTemplate
405+ 
406+ 略
407+ 
408+- ##### cluster-api-provider-docker（Infrastructure Provider）
409+ 
410+ - DockerCluster
411+ 
412+ ```yaml
413+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
414+ kind: DockerCluster
415+ metadata:
416+ name: my-cluster
417+ namespace: default
418+ ```
419+ 
420+ - DockerMachineTemplate
421+ 
422+ ```yaml
423+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
424+ kind: DockerMachineTemplate
425+ metadata:
426+ name: controlplane
427+ namespace: default
428+ spec:
429+ template:
430+ spec: {}
431+ ---
432+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
433+ kind: DockerMachineTemplate
434+ metadata:
435+ name: worker
436+ namespace: default
437+ spec:
438+ template:
439+ spec: {}
440+ ```
441+ 
442+ - DockerMachine
443+ 
444+ 由DockerMachineTemplate自动填充字段
445+ 
446+**综上所述，一个基础的Target Cluster yaml，必须包含 Cluster、MachineDeployment、KubeadmControlPlane、KubeadmConfigTemplate（worker）、infrastructureMachineTemplate和infrastructureCluster共计6个CRD**
447+ 
448+#### CRD关系图
449+ 
450+上面的CRD关系复杂看不明白，没关系一图看懂Cluster API CRD关系
451+ 
452+[官方CRD关系图](https://cluster-api.sigs.k8s.io/developer/crd-relationships.html)
453+ 
454+![](../image/cluster-api-v3-image/Cluster%20API%20CRD.drawio.svg)
455+ 
456+#### 基于cluster-api-provider-docker的Cluster-api工作流程
457+ 
458+##### 名词解释
459+ 
460+- cluster-api-provider-docker
461+ 
462+ 是将集群部署在本地docker容器中的基础设施提供者
463+ 
464+- 管理集群
465+ 
466+ 是Cluster API及其相关组件所在的集群
467+ 
468+- Target Cluster
469+ 
470+ 是受Cluster API 管理的集群
471+ 
472+ 
473+ 
474+##### 安装clusterctl
475+ 
476+###### 先决条件
477+ 
478+- 安装kind和docker
479+- 安装kubectl
480+ 
481+###### 安装
482+ 
483+```bash
484+curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.1.3/clusterctl-linux-amd64 -o clusterctl
485+ 
486+chmod +x ./clusterctl
487+ 
488+sudo mv ./clusterctl /usr/local/bin/clusterctl
489+ 
490+clusterctl version
491+```
492+ 
493+###### 使用kind创建管理集群
494+ 
495+```bash
496+cat <<EOF > kind_cluster_config.yaml
497+kind: Cluster
498+apiVersion: kind.x-k8s.io/v1alpha4
499+nodes:
500+ - role: control-plane
501+ extraMounts:
502+ - hostPath: /var/run/docker.sock
503+ containerPath: /var/run/docker.sock
504+EOF
505+ 
506+kind create cluster --config ./kind_cluster_config.yaml
507+```
508+ 
509+###### 初始化Cluster API
510+ 
511+```bash
512+# 设置Infrastructure Provider为docker
513+clusterctl init --infrastructure docker
514+```
515+ 
516+第一次执行时，`clusterctl init` 会自动将核心提供程序添加到列表中`cluster-api`，如果未指定，它会添加`kubeadm-bootstrap provider`和`kubeadm-control-plane provider`为默认的Bootstrap provider和Control Plane provider。
517+ 
518+###### 有关 `clusterctl init`
519+ 
520+- clusterctl init 是Cluster API初始化管理集群的命令
521+ 
522+- 在初始化过程中会按照`$HOME/.cluster-api/clusterctl.yaml`文件或者默认的配置从Github下载所有有关Provider 的包含Provider `CRD`、`Role`等资源的yaml文件，并注册到管理集群中。
523+ 
524+- 可通过 `clusterctl config repositories`查看可以使用的Provider
525+ 
526+ ![](../image/cluster-api-v3-image/config-respositories.png)
527+ 
528+##### 部署docker 集群
529+ 
530+场景：部署一个三主三从的k8s集群
531+ 
532+- ###### 使用clusterctl 生成集群配置yaml
533+ 
534+ ```sh
535+ clusterctl generate cluster capi-quickstart --flavor development \
536+ --kubernetes-version v1.23.4 \
537+ --control-plane-machine-count=3 \
538+ --worker-machine-count=3 \
539+ > capi-quickstart.yaml
540+ ```
541+ 
542+- ###### 提交yaml到管理集群
543+ 
544+ ```sh
545+ kubectl apply -f capi-quickstart.yaml
546+ ```
547+ 
548+ 输出
549+ 
550+ ```tex
551+ cluster.cluster.x-k8s.io/capi-quickstart created
552+ dockercluster.infrastructure.cluster.x-k8s.io/capi-quickstart created
553+ kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane created
554+ dockermachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane created
555+ machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 created
556+ dockermachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 created
557+ kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 created
558+ ```
559+ 
560+##### 组件工作流程
561+ 
562+###### Target Cluster证书生成
563+ 
564+证书的生成由kubeadm-control-plane-provider组件实现
565+ 
566+- 检测当前namespace下是否存在储存了证书数据的secret
567+ 
568+- 不存在则依次创建集群的ca.crt、ca.key、sa.pub、sa.key、front-proxy-ca.crt、front-proxy-ca.key，如果yaml（KubeadmControlPlane CRD）中未提供etcd的外部证书则还会生成etcd的ca.crt、ca.key
569+ 
570+ 注：证书私钥都是随机生成的
571+ 
572+- 将生成的证书数据写入到同Namespace下的Secret中（XXX-ca，XXX-sa, XXX-etcd, XXX-proxy）
573+ 
574+ ![](../image/cluster-api-v3-image/cluster%20secret.png)
575+ 
576+- 根据生成的XXX-ca证书生成Target Cluster的kube.config数据并写入到同Namespace下的Secret中XXX-kubeconfig
577+ 
578+ 
579+ 
580+###### Kubeadm命令生成
581+ 
582+由kubeadm-bootstrap-provider组件实现
583+ 
584+- KubeadmConfig对象通过OwnerReferences获取关联的Machine
585+ 
586+- 如果Machine是Control Plane则创建且只创建一次control plane的`kubeadm init` 配置文件kubeadm.yaml
587+ 
588+- 将kubeadm.yaml数据写入到同Namespace下的Secret中
589+ 
590+ ![image-20220429104540249](../image/cluster-api-v3-image/data secret.png)
591+ 
592+- 将secret name 同步到关联Machine的 .Status.DataSecret
593+ 
594+- 后续的协调中则根据关联的Machine是否为control plane重复上述步骤生成 control plane join 或者 worker join 的`kubeadm join`的kubeadm.yaml
595+ 
596+- 有关kubeadm.yaml数据的生成
597+ 
598+ - 值得一提的是在CRD中对kubeadm的配置与kubeadm独立使用时的配置一致
599+ 
600+ - 在yaml（KubeadmControlPlane CRD）的配置时
601+ 
602+ spec.kubeadmConfigSpec.clusterConfiguration和spec.kubeadmConfigSpec.initConfiguration字段会被写入到control plane第一次init的kubeadm.yaml中
603+ 
604+ spec.kubeadmConfigSpec.joinConfiguration字段会被用于后续其他control plane 节点加入的kubeadm.yaml中
605+ 
606+ - 在yaml（KubeadmConfigTemplate CRD）
607+ 
608+ template.sepc.joinConfiguration字段会被用于worker节点加入的kubeadm.yaml中
609+ 
610+ - 储存kubeadm.yaml的Secret并不是简单的yaml格式，在这里cluster-api使用了Cloud-Init的格式将集群相关证书和kubeadm.yaml的内容以及相关kubeadm执行命令一并存储在该Secret中
611+ 
612+ ```yaml
613+ ## template: jinja
614+ #cloud-config
615+
616+ write_files:
617+
618+ ......
619+
620+ - path: /etc/kubernetes/pki/sa.key
621+ owner: root:root
622+ permissions: '0600'
623+ content: |
624+ <replace-with-a-newly-generated-private-key>
625+
626+ - path: /run/kubeadm/kubeadm-join-config.yaml
627+ owner: root:root
628+ permissions: '0640'
629+ content: |
630+ apiVersion: kubeadm.k8s.io/v1beta3
631+ controlPlane:
632+ localAPIEndpoint: {}
633+ discovery:
634+ bootstrapToken:
635+ apiServerEndpoint: 172.18.0.3:6443
636+ caCertHashes:
637+ - sha256:<ca-cert-public-key-hash>
638+ token: <bootstrap-token>
639+ kind: JoinConfiguration
640+ nodeRegistration:
641+ criSocket: /var/run/containerd/containerd.sock
642+ kubeletExtraArgs:
643+ cgroup-driver: cgroupfs
644+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
645+ taints: null
646+
647+ .....
648+
649+ runcmd:
650+ - kubeadm join --config /run/kubeadm/kubeadm-join-config.yaml && echo success > /run/cluster-api/bootstrap-success.complete
651+ ```
652+ 
653+ 
654+ 
655+ 
656+###### 删除集群
657+ 
658+- ```
659+ kubectl delete Cluster capi-quickstart -n foo
660+ ```
661+ 
662+ 或者
663+ 
664+ ```
665+ kubectl delete -f capi-quickstart.yaml
666+ ```
667+ 
668+注意：由于Cluster API自身的原因，若集群创建中出错，且provider未能处理好这些错误。此时若想从管理集群删除Cluster及其子孙资源，可能会卡住无法删除，暂时没有很好的方法解决
669+ 
670+ 
671+ 
672+###### 查看集群创建情况
673+ 
674+- 查看集群control plane创建情况
675+ 
676+ ```bash
677+ kubectl get kubeadmcontrolplane -n foo
678+ ```
679+ 
680+ 输出如下
681+ 
682+ ![image-20220429153602723](../image/cluster-api-v3-image/get controlplane.png)
683+ 
684+- 查看集群worker创建情况
685+ 
686+ ```bash
687+ kubectl get machinedeployment -n foo
688+ ```
689+ 
690+ 输出如下
691+ 
692+ ![image-20220429153804948](../image/cluster-api-v3-image/get worker.png)
693+ 
694+- 使用clusterctl查看集群
695+ 
696+ ```bash
697+ clusterctl describe cluster capi-quickstart --namespace foo
698+ ```
699+ 
700+ 输出如下
701+ 
702+ ![](../image/cluster-api-v3-image/clusterctl describle.png)
703+ 
704+###### 获取Target Cluster kubeconfig
705+ 
706+- 使用clusterctl 获取Target Cluster kubeconfig并保存
707+ 
708+ ```bash
709+ clusterctl get kubeconfig capi-quickstart --namespace foo > ./capi-quickstar-kube.config
710+ ```
711+ 
712+##### Cluster API搭建高可用集群流程图
713+ 
714+![](../image/cluster-api-v3-image/cluster api workflow.png)
715+ 
716+#### 可能存在的问题或者缺陷
717+ 
718+- 项目结构复杂
719+- 集群创建过程中出错可能会导致无法删除Cluster及关联资源
720+- CRD设计复杂衔接紧密
721+ 
722+#### cluster-api-provider-***设计注意事项
723+ 
724+对于自定义provider，必须要拟定两个CRD和对应的controller（demoCluster，demoMachine）
725+ 
726+##### DemoCluster
727+ 
728+DemoCluster提供运行机器所需的任何先决条件。示例可能包括网络、负载平衡器、防火墙规则等。
729+ 
730+- 使用时必须与Cluster API 所属同一命名空间
731+ 
732+- Spec.ControlPlaneEndpoint（必须）
733+ 
734+ - 该字段必须设置，由controller自动填充，是集群控制平面负载均衡器的信息
735+ 
736+ - 字段被设置后会被Cluster 控制器读取并填充到Cluster.spec.ControlPlaneEndpoint字段中，后续Cluster API对Target Cluster的API Server的访问都依据此字段
737+ 
738+ - Target Cluster的kubeconfig 的生成也会依赖此字段进行设置
739+ 
740+ ```go
741+ type APIEndpoint struct {
742+ // Host is the hostname on which the API server is serving.
743+ Host string `json:"host"`
744+
745+ // Port is the port on which the API server is serving.
746+ Port int `json:"port"`
747+ }
748+ ```
749+ 
750+- Status.Ready（boolean）（必须）
751+ 
752+ 在demoClusterCRD中，必须设置Status.Ready字段。只有Ready为true时，Cluster API的后续工作流程才能正常运行。
753+ 
754+ - 什么时候设置为true
755+ 
756+ 从目前调研的情况来看democluster的主要功能是配置负载均衡器，并将负载均衡器的地址和端口填充到demoCluster.Spec.ControlPlaneEndpoint中即可。所以在确认负载均衡器已经配置好且启动后可将demoCluster.Status.Ready设置为true
757+ 
758+- 可选字段
759+ 
760+ 参考[官网解释](https://cluster-api.sigs.k8s.io/developer/providers/cluster-infrastructure.html#infracluster-resources)
761+ 
762+另外对于DemoCluster还需要搞一个DemoClusterTemplate CRD，格式参考如下
763+ 
764+```go
765+// InfraClusterTemplateSpec defines the desired state of InfraClusterTemplate.
766+type InfraClusterTemplateSpec struct {
767+ Template InfraClusterTemplateResource `json:"template"`
768+}
769+ 
770+// +kubebuilder:object:root=true
771+// +kubebuilder:resource:path=infraclustertemplates,scope=Namespaced,categories=cluster-api,shortName=ict
772+// +kubebuilder:storageversion
773+ 
774+// InfraClusterTemplate is the Schema for the infraclustertemplates API.
775+type InfraClusterTemplate struct {
776+ metav1.TypeMeta `json:",inline"`
777+ metav1.ObjectMeta `json:"metadata,omitempty"`
778+ 
779+ Spec InfraClusterTemplateSpec `json:"spec,omitempty"`
780+}
781+ 
782+type InfraClusterTemplateResource struct {
783+ // Standard object's metadata.
784+ // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
785+ // +optional
786+ ObjectMeta clusterv1.ObjectMeta `json:"metadata,omitempty"`
787+ Spec InfraClusterSpec `json:"spec"`
788+}
789+```
790+ 
791+##### DemoMachine
792+ 
793+DemoMachine是对Target Cluster 节点的实际提供者。
794+ 
795+- 使用时必须与Cluster API 所属同一命名空间
796+ 
797+- 若使用kubeadm bootstrap provider作为集群引导程序，则需要对kubeadm bootstrap provider生成的包含kubeadm init || join等命令的Secret进行解析，并将命令在实际的虚机或者裸机上执行。
798+ 
799+- 在管理集群对Target Cluster进行删除操作时，应当对实际的机器进行还原（kubeadm reset），如有必要并将机器关机，如果是虚机应当释放该虚机所占资源。
800+ 
801+- 当DemoMachine关联的Machine资源lable标记有`cluster.x-k8s.io/control-plane: ""` 时，除执行kubeadm命令还需要将实际的节点流量交由负载均衡器控制
802+ 
803+- Spec.providerID（string）（必须）
804+ 
805+ 该字段必须设置由DemoMachine controller自动填充。
806+ 
807+ - 什么时候设置
808+ 
809+ 从目前的调研情况来看，该字段应当在确认实际的机器执行kubeadm init|join命令成功后设置。是一个由controller拟定的唯一值
810+
811+ - 设置的格式
812+ 
813+ 官方文档并没有给出providerID的具体格式，但在实际demo使用中Cluster API是要求了具体格式的，如下
814+
815+ ```
816+ <cloudProvider>://<optional>/<segments>/<provider id>
817+ #示例
818+ # docker provider
819+ docker:////capi-quickstart-control-plane-85h47
820+
821+ # thinkerbell provider
822+ tinkerbell://[hardware.Spec.ID]
823+
824+ # metal3 provider
825+ metal3://[bmhID]
826+ ```
827+
828+ 也可参考Node.Spec.ProviderID
829+
830+ ```go
831+ // ID of the node assigned by the cloud provider in the format: <ProviderName>://<ProviderSpecificNodeID>
832+ // +optional
833+ ProviderID string `json:"providerID,omitempty" protobuf:"bytes,3,opt,name=providerID"`
834+ ```
835+
836+
837+
838+ - 特別注意
839+ 
840+ 该值不仅仅是直接填充后就结束，应当还需要在对应的Target cluster的node资源上标记该值，后续Cluster API的Machine controller会根据该值去匹配对应的Target cluster的node，并设置NodeRef与该node资源关联，在Status存储该node的nodeInfo
841+ 
842+ 若设置的providerId不正确或者未在实际的Target cluster的node资源上标记ProviderID，并不会影响Target cluster的搭建，只是在Machine资源上不能看到对应Target cluster的node的信息
843+ 
844+ ![](../image/cluster-api-v3-image/DemoMachine注意事项.png)
845+ 
846+- Status.Ready （boolean）（必须）
847+ 
848+ 在DemoMachine CRD中，必须设置Status.Ready字段。只有Ready为true时，Cluster API的后续工作流程才能正常运行。
849+ 
850+ - 什么时候设置
851+ 
852+ 同providerID，该字段应当在确认实际的机器执行kubeadm init|join命令成功后设置为ture
853+ 
854+- 可选字段
855+ 
856+ 参考[官网解释](https://cluster-api.sigs.k8s.io/developer/providers/machine-infrastructure.html#data-types)
857+ 
858+另外对于DemoMachine还需要搞一个DemoMachineTemplate CRD，格式参考如下
859+ 
860+```go
861+// InfraMachineTemplateSpec defines the desired state of InfraMachineTemplate.
862+type InfraMachineTemplateSpec struct {
863+ Template InfraMachineTemplateResource `json:"template"`
864+}
865+ 
866+// +kubebuilder:object:root=true
867+// +kubebuilder:resource:path=inframachinetemplates,scope=Namespaced,categories=cluster-api,shortName=imt
868+// +kubebuilder:storageversion
869+ 
870+// InfraMachineTemplate is the Schema for the inframachinetemplates API.
871+type InfraMachineTemplate struct {
872+ metav1.TypeMeta `json:",inline"`
873+ metav1.ObjectMeta `json:"metadata,omitempty"`
874+ 
875+ Spec InfraMachineTemplateSpec `json:"spec,omitempty"`
876+}
877+ 
878+type InfraMachineTemplateResource struct {
879+ // Standard object's metadata.
880+ // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
881+ // +optional
882+ ObjectMeta clusterv1.ObjectMeta `json:"metadata,omitempty"`
883+ Spec InfraMachineSpec `json:"spec"`
884+}
885+```
Adesign/BKE设计文档/cluster-api/cluster-api原理分析.pdf+0-0
Binary files do not support preview
Adesign/BKE设计文档/cluster-api/k8s证书续期管理机制.md+15-0
@@ -0,0 +1,15 @@
1+- Cluster API 目前不支持对集群证书管理，默认生成的有效期十年无法从其他配置处修改。
2+ 
3+ 相关issue
4+ 
5+ - https://github.com/kubernetes-sigs/cluster-api/issues/6529
6+ 
7+ 
8+ - https://github.com/kubernetes-sigs/cluster-api/issues/5490
9+ 
10+ 
11+ - https://github.com/kubernetes-sigs/cluster-api/issues/5435
12+ 
13+ 
14+- Cluster API只负责集群相关证书(etcd，proxy，ca等)生成，对证书生命周期不作管理
15+ 
Adesign/BKE设计文档/cluster-api/kube-apiserver可选参数配置.md+48-0
@@ -0,0 +1,48 @@
1+- APIServer的配置出现在KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.apiServer
2+ 
3+ ```go
4+ // APIServer holds settings necessary for API server deployments in the cluster.
5+ type APIServer struct {
6+ ControlPlaneComponent `json:",inline"`
7+
8+ // CertSANs sets extra Subject Alternative Names for the API Server signing cert.
9+ // +optional
10+ CertSANs []string `json:"certSANs,omitempty"`
11+
12+ // TimeoutForControlPlane controls the timeout that we use for API server to appear
13+ // +optional
14+ TimeoutForControlPlane *metav1.Duration `json:"timeoutForControlPlane,omitempty"`
15+ }
16+ ```
17+ 
18+
19+ 
20+- APIServer配置示例
21+ 
22+ 有关APIServer详细配置，请参阅 [kube-apiserver 参考文档](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/)
23+ 
24+ ```yaml
25+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
26+ kind: KubeadmControlPlane
27+ metadata:
28+ name: controlplane
29+ namespace: default
30+ spec:
31+
32+ ......
33+
34+ kubeadmConfigSpec:
35+ clusterConfiguration:
36+ apiServer:
37+ certSANs:
38+ - localhost
39+ - 127.0.0.1
40+ - 0.0.0.0
41+ extraArgs:
42+ anonymous-auth: "false"
43+ enable-admission-plugins: AlwaysPullImages,DefaultStorageClass
44+ audit-log-path: /home/johndoe/audit.log
45+ ......
46+
47+ ......
48+ ```
Adesign/BKE设计文档/cluster-api/kube-controller可选参数配置.md+52-0
@@ -0,0 +1,52 @@
1+- controllerManager的配置出现在KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.controllerManager
2+ 
3+ ```go
4+ // ControlPlaneComponent holds settings common to control plane component of the cluster.
5+ type ControlPlaneComponent struct {
6+ // ExtraArgs is an extra set of flags to pass to the control plane component.
7+ // TODO: This is temporary and ideally we would like to switch all components to
8+ // use ComponentConfig + ConfigMaps.
9+ // +optional
10+ ExtraArgs map[string]string `json:"extraArgs,omitempty"`
11+
12+ // ExtraVolumes is an extra set of host volumes, mounted to the control plane component.
13+ // +optional
14+ ExtraVolumes []HostPathMount `json:"extraVolumes,omitempty"`
15+ }
16+ ```
17+ 
18+- controllerManager配置示例
19+ 
20+ 有关controllerManager详细配置信息，请参阅 [kube-controller-manager 参考文档](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/)
21+ 
22+ ```yaml
23+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
24+ kind: KubeadmControlPlane
25+ metadata:
26+ name: controlplane
27+ namespace: default
28+ spec:
29+
30+ ......
31+
32+ kubeadmConfigSpec:
33+ clusterConfiguration:
34+ controllerManager:
35+ extraArgs:
36+ enable-hostpath-provisioner: "true"
37+ cluster-signing-key-file: /home/johndoe/keys/ca.key
38+ deployment-controller-sync-period: "50"
39+ ......
40+ extraVolumes:
41+ - name: controllerconfig
42+ hostPath: /home/johndoe/controller.yaml
43+ mountPath: /etc/kubernetes/controller-manager-config.yaml
44+ readOnly: true
45+ pathType: "File"
46+ ......
47+
48+ ......
49+ ```
50+
51+
52+ 
Adesign/BKE设计文档/cluster-api/kube-scheduler可选参数配置.md+49-0
@@ -0,0 +1,49 @@
1+- Scheduler的配置出现在KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.scheduler
2+ 
3+ ```go
4+ type ControlPlaneComponent struct {
5+ // ExtraArgs is an extra set of flags to pass to the control plane component.
6+ // TODO: This is temporary and ideally we would like to switch all components to
7+ // use ComponentConfig + ConfigMaps.
8+ // +optional
9+ ExtraArgs map[string]string `json:"extraArgs,omitempty"`
10+
11+ // ExtraVolumes is an extra set of host volumes, mounted to the control plane component.
12+ // +optional
13+ ExtraVolumes []HostPathMount `json:"extraVolumes,omitempty"`
14+ }
15+ ```
16+ 
17+- Scheduler配置示例
18+ 
19+ 有关Scheduler详细配置，请参阅 [kube-scheduler 参考文档](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/)
20+ 
21+ ```yaml
22+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
23+ kind: KubeadmControlPlane
24+ metadata:
25+ name: controlplane
26+ namespace: default
27+ spec:
28+
29+ ......
30+
31+ kubeadmConfigSpec:
32+ clusterConfiguration:
33+ scheduler:
34+ extraArgs:
35+ config: /etc/kubernetes/scheduler-config.yaml
36+ ......
37+ extraVolumes:
38+ - name: schedulerconfig
39+ hostPath: /home/johndoe/schedconfig.yaml
40+ mountPath: /etc/kubernetes/scheduler-config.yaml
41+ readOnly: true
42+ pathType: "File"
43+ ......
44+
45+ ......
46+ ```
47+ 
48+
49+ 
Adesign/BKE设计文档/cluster-api/kubeadm可选参数配置.md+142-0
@@ -0,0 +1,142 @@
1+在Cluster API 中kubeadm的配置文件kubeadm.yml由Cluster API的kubeadm-bootstrap-provider组件生成
2+ 
3+Cluster API中对kubeadm的配置目前只会生成ClusterConfiguration、InitConfiguration、JoinConfiguration配置类型的yml数据
4+ 
5+参考[kubeadm配置](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/)
6+ 
7+- 集群范围配置`ClusterConfiguration`
8+ 
9+ ClusterConfiguration配置在KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration下
10+ 
11+ 参考[ClusterConfiguration详细配置](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-ClusterConfiguration)
12+ 
13+ 可选配置如下：
14+ 
15+ - etcd
16+ 
17+ 详情见[etcd服务配置](https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/control-plane-flags/#etcd-flags)
18+ 
19+ - networking
20+ 
21+ 该字段通常在Cluster资源对象中配置，会被自动填写到kubeadm.yaml中的networking字段
22+ 
23+ ```
24+ apiVersion: cluster.x-k8s.io/v1beta1
25+ kind: Cluster
26+ metadata:
27+ name: demo-cluster
28+ namespace: demo-cluster
29+ spec:
30+ clusterNetwork:
31+ services:
32+ cidrBlocks: ["10.96.0.0/12"]
33+ pods:
34+ cidrBlocks: ["192.168.0.0/16"]
35+ serviceDomain: "cluster.local"
36+ ```
37+ 
38+ 详情见[集群的网络拓扑配置](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-Networking)
39+ 
40+ - kubernetesVersion
41+ 
42+ 该值默认为 Machine 对象 spec.version
43+ 
44+ - apiServer
45+ 
46+ [kube-apiserver可选参数配置](kube-apiserver可选参数配置.md)
47+ 
48+ - controllerManager
49+ 
50+ [kube-controller可选参数配置](kube-controller可选参数配置.md)
51+ 
52+ - scheduler
53+ 
54+ [kube-scheduler可选参数配置](kube-scheduler可选参数配置.md)
55+ 
56+ - dns
57+ 
58+ `dns` 定义在集群中安装的 DNS 插件的选项。详情见[集群DNS插件配置](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-DNS)
59+ 
60+ - certificatesDir
61+ 
62+ 设置在何处存放或者查找所需证书
63+ 
64+ - imageRepository
65+ 
66+ `imageRepository` 设置用来拉取镜像的容器仓库。 如果此字段为空，默认使用 `k8s.gcr.io`。 当 Kubernetes 用来执行 CI 构造时（Kubernetes 版本以 `ci/` 开头）， 将默认使用 `gcr.io/k8s-staging-ci-images` 来拉取控制面组件镜像， 而使用 `k8s.gcr.io` 来拉取所有其他镜像。
67+ 
68+ - featureGates
69+ 
70+ `featureGates` 包含用户所启用的特性门控。
71+ 
72+ - clusterName
73+ 
74+ 集群名称
75+ 
76+- control plane初始化配置`InitConfiguration`
77+ 
78+ InitConfiguration配置在KubeadmControlPlane.spec.kubeadmConfigSpec.InitConfiguration下
79+ 
80+ 参考[InitConfiguration详细配置](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-InitConfiguration)
81+ 
82+ 可选配置如下：
83+ 
84+ - bootstrapTokens
85+ 
86+ 可以忽略，Cluster API的生成的cloud-init配置中已携带token等信息，交由Infrastructure provider在具体机器上进行持久化
87+ 
88+ - nodeRegistration
89+ 
90+ 集群节点的配置，可在此处单独配置该节点的kubelet，运行时等
91+ 
92+ 详见[集群中注册新的控制面节点相关的字段](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-NodeRegistrationOptions)
93+ 
94+ - localAPIEndpoint
95+ 
96+ 代表的的是在此控制面节点上要部署的 API 服务器 的端点。在高可用（HA）配置中，此字段与 `ClusterConfiguration.controlPlaneEndpoint` 的取值不同：后者代表的是整个集群的全局端点，该端点上的请求会被负载均衡到每个 API 服务器。 此配置对象允许你定制本地 API 服务器所公布的、可访问的 IP/DNS 名称和端口。 默认情况下，kubeadm 会尝试自动检测默认接口上的 IP 并使用该地址。 不过，如果这种检测失败，你可以在此字段中直接设置所期望的值。
97+ 
98+ - certificateKey
99+ 
100+ `certificateKey` 用来设置一个秘钥，该秘钥将对 `uploadcerts init` 阶段上传到集群中某 Secret 内的秘钥和证书加密。
101+ 
102+ - skipPhases
103+ 
104+ `skipPhases` 是命令执行过程中药略过的阶段（Phases）。 通过执行命令 `kubeadm init --help` 可以获得阶段的列表。 参数标志 "--skip-phases" 优先于此字段的设置。
105+ 
106+ - patches
107+ 
108+ `patches` 包含与 `kubeadm init` 阶段 kubeadm 所部署 的组件上要应用的补丁相关的信息。
109+ 
110+- control plane || worker节点加入配置`JoinConfiguration`
111+ 
112+ JoinConfiguration配置在KubeadmControlPlane.spec.kubeadmConfigSpec.JoinConfiguration下，通常是对control plane节点加入的配置，另外也会出现在KubeadmConfigTemplate.spec.template.spec.joinConfiguration中，提供worker节点加入的配置
113+ 
114+ 参考[JoinConfiguration详细配置](https://kubernetes.io/zh/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-JoinConfiguration)
115+ 
116+ 可选配置如下：
117+ 
118+ - nodeRegistration
119+ 
120+ 同InitConfiguration中的nodeRegistration配置
121+ 
122+ - caCertPath
123+ 
124+ `caCertPath` 是指向 SSL 证书机构的路径，该证书包用来加密 节点与控制面之间的通信。默认值为 "/etc/kubernetes/pki/ca.crt"。
125+ 
126+ ps：可以忽略配置，Cluster API的生成的cloud-init配置中已携带token等信息，交由Infrastructure provider在具体机器上进行持久化
127+ 
128+ - discovery
129+ 
130+ `discovery` 设置 TLS 引导过程中 kubelet 要使用的选项。
131+ 
132+ - controlPlane
133+ 
134+ `controlPlane` 定义要在正被加入到集群中的节点上部署的额外 控制面实例。此字段为 null 时，不会再上面部署额外的控制面实例。
135+ 
136+ - skipPhases
137+ 
138+ 此字段包含在命令执行过程中要略过的阶段。通过 `kubeadm join --help` 命令可以查看阶段的列表。参数 `--skip-phases` 优先于此字段。
139+ 
140+ - patches
141+ 
142+ 此字段包含 `kubeadm join` 阶段向 kubeadm 所部署的组件打补丁 的选项
Adesign/BKE设计文档/cluster-api/kubelet可选参数配置.md+46-0
@@ -0,0 +1,46 @@
1+- Cluster API 支持对每个节点配置kubelet
2+ 
3+ - control plane 节点配置kubelet
4+ 
5+ - control plane 初始化第一个节点时配置kubectl
6+ 
7+ KubeadmControlPlane.spec.kubeadmConfigSpec.**initConfiguration**.nodeRegistration.kubeletExtraArgs
8+ 
9+ - control plane 其他节点加入时配置kubectl
10+ 
11+ KubeadmControlPlane.spec.kubeadmConfigSpec.**joinConfiguration**.nodeRegistration.kubeletExtraArgs
12+ 
13+ - worker 节点配置kubectl
14+ 
15+ KubeadmConfigTemplate.spec.template.spec.joinConfiguration.nodeRegistration.kubeletExtraArgs
16+ 
17+- kubeletExtraArgs详细配置说明
18+ 
19+ - 在使用该参数前请查阅[kubectl参考文档](https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kubelet/)，有些kubelet参数或许已经弃用
20+ 
21+ - 配置说明，由于kubelet可配置参数过多，详细参数配置可查阅[kubectl参考文档](https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kubelet/)
22+ 
23+ ```go
24+ KubeletExtraArgs map[string]string `json:"kubeletExtraArgs,omitempty"`
25+ ```
26+ 
27+ 使用时，以worker节点配置kubelet为例
28+ 
29+ ```yaml
30+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
31+ kind: KubeadmConfigTemplate
32+ metadata:
33+ name: worker
34+ spec:
35+ template:
36+ spec:
37+ joinConfiguration:
38+ nodeRegistration:
39+ kubeletExtraArgs:
40+ cgroup-driver: cgroupfs
41+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
42+ ......
43+ ```
44+ 
45+
46+ 
Adesign/BKE设计文档/cluster-api/单主多节点.yaml+129-0
@@ -0,0 +1,129 @@
1+# 以docker provider为例
2+# 注意MachineDeployment和KubeadmControlPlane的replicas字段值
3+ 
4+# 创建一个1主 3节点的集群
5+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
6+kind: DockerCluster
7+metadata:
8+ name: my-cluster
9+ namespace: default
10+---
11+apiVersion: cluster.x-k8s.io/v1beta1
12+kind: Cluster
13+metadata:
14+ name: my-cluster
15+ namespace: default
16+spec:
17+ clusterNetwork:
18+ services:
19+ cidrBlocks: ["10.96.0.0/12"]
20+ pods:
21+ cidrBlocks: ["192.168.0.0/16"]
22+ serviceDomain: "cluster.local"
23+ controlPlaneRef:
24+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
25+ kind: KubeadmControlPlane
26+ name: controlplane
27+ namespace: default
28+ infrastructureRef:
29+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
30+ kind: DockerCluster
31+ name: my-cluster
32+ namespace: default
33+---
34+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
35+kind: DockerMachineTemplate
36+metadata:
37+ name: controlplane
38+ namespace: default
39+spec:
40+ template:
41+ spec: {}
42+---
43+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
44+kind: KubeadmControlPlane
45+metadata:
46+ name: controlplane
47+ namespace: default
48+spec:
49+ replicas: 1
50+ version: v1.23.3
51+ machineTemplate:
52+ infrastructureRef:
53+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
54+ kind: DockerMachineTemplate
55+ name: controlplane
56+ namespace: default
57+ kubeadmConfigSpec:
58+ clusterConfiguration:
59+ apiServer:
60+ certSANs:
61+ - localhost
62+ - 127.0.0.1
63+ - 0.0.0.0
64+ controllerManager:
65+ extraArgs:
66+ enable-hostpath-provisioner: "true"
67+ initConfiguration:
68+ nodeRegistration:
69+ kubeletExtraArgs:
70+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
71+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
72+ cgroup-driver: cgroupfs
73+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
74+ joinConfiguration:
75+ nodeRegistration:
76+ kubeletExtraArgs:
77+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
78+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
79+ cgroup-driver: cgroupfs
80+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
81+---
82+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
83+kind: DockerMachineTemplate
84+metadata:
85+ name: worker
86+ namespace: default
87+spec:
88+ template:
89+ spec: {}
90+---
91+apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
92+kind: KubeadmConfigTemplate
93+metadata:
94+ name: worker
95+spec:
96+ template:
97+ spec:
98+ joinConfiguration:
99+ nodeRegistration:
100+ kubeletExtraArgs:
101+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
102+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
103+ cgroup-driver: cgroupfs
104+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
105+---
106+apiVersion: cluster.x-k8s.io/v1beta1
107+kind: MachineDeployment
108+metadata:
109+ name: worker-md-0
110+spec:
111+ clusterName: my-cluster
112+ replicas: 3
113+ selector:
114+ matchLabels:
115+ cluster.x-k8s.io/cluster-name: my-cluster
116+ template:
117+ spec:
118+ version: v1.23.3
119+ clusterName: my-cluster
120+ bootstrap:
121+ configRef:
122+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
123+ kind: KubeadmConfigTemplate
124+ name: worker
125+ infrastructureRef:
126+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
127+ kind: DockerMachineTemplate
128+ name: worker
129+ 
Adesign/BKE设计文档/cluster-api/多主多节点.yaml+129-0
@@ -0,0 +1,129 @@
1+# 以docker provider为例
2+# 注意MachineDeployment和KubeadmControlPlane的replicas字段值
3+ 
4+# 创建一个 3主 3节点的集群
5+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
6+kind: DockerCluster
7+metadata:
8+ name: my-cluster
9+ namespace: default
10+---
11+apiVersion: cluster.x-k8s.io/v1beta1
12+kind: Cluster
13+metadata:
14+ name: my-cluster
15+ namespace: default
16+spec:
17+ clusterNetwork:
18+ services:
19+ cidrBlocks: ["10.96.0.0/12"]
20+ pods:
21+ cidrBlocks: ["192.168.0.0/16"]
22+ serviceDomain: "cluster.local"
23+ controlPlaneRef:
24+ apiVersion: controlplane.cluster.x-k8s.io/v1beta1
25+ kind: KubeadmControlPlane
26+ name: controlplane
27+ namespace: default
28+ infrastructureRef:
29+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
30+ kind: DockerCluster
31+ name: my-cluster
32+ namespace: default
33+---
34+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
35+kind: DockerMachineTemplate
36+metadata:
37+ name: controlplane
38+ namespace: default
39+spec:
40+ template:
41+ spec: {}
42+---
43+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
44+kind: KubeadmControlPlane
45+metadata:
46+ name: controlplane
47+ namespace: default
48+spec:
49+ replicas: 3
50+ version: v1.23.3
51+ machineTemplate:
52+ infrastructureRef:
53+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
54+ kind: DockerMachineTemplate
55+ name: controlplane
56+ namespace: default
57+ kubeadmConfigSpec:
58+ clusterConfiguration:
59+ apiServer:
60+ certSANs:
61+ - localhost
62+ - 127.0.0.1
63+ - 0.0.0.0
64+ controllerManager:
65+ extraArgs:
66+ enable-hostpath-provisioner: "true"
67+ initConfiguration:
68+ nodeRegistration:
69+ kubeletExtraArgs:
70+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
71+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
72+ cgroup-driver: cgroupfs
73+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
74+ joinConfiguration:
75+ nodeRegistration:
76+ kubeletExtraArgs:
77+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
78+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
79+ cgroup-driver: cgroupfs
80+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
81+---
82+apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
83+kind: DockerMachineTemplate
84+metadata:
85+ name: worker
86+ namespace: default
87+spec:
88+ template:
89+ spec: {}
90+---
91+apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
92+kind: KubeadmConfigTemplate
93+metadata:
94+ name: worker
95+spec:
96+ template:
97+ spec:
98+ joinConfiguration:
99+ nodeRegistration:
100+ kubeletExtraArgs:
101+ # We have to pin the cgroupDriver to cgroupfs as kubeadm >=1.21 defaults to systemd
102+ # kind will implement systemd support in: https://github.com/kubernetes-sigs/kind/issues/1726
103+ cgroup-driver: cgroupfs
104+ eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
105+---
106+apiVersion: cluster.x-k8s.io/v1beta1
107+kind: MachineDeployment
108+metadata:
109+ name: worker-md-0
110+spec:
111+ clusterName: my-cluster
112+ replicas: 3
113+ selector:
114+ matchLabels:
115+ cluster.x-k8s.io/cluster-name: my-cluster
116+ template:
117+ spec:
118+ version: v1.23.3
119+ clusterName: my-cluster
120+ bootstrap:
121+ configRef:
122+ apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
123+ kind: KubeadmConfigTemplate
124+ name: worker
125+ infrastructureRef:
126+ apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
127+ kind: DockerMachineTemplate
128+ name: worker
129+ 
Adesign/BKE设计文档/cluster-api/如何伸缩k8s节点.md+8-0
@@ -0,0 +1,8 @@
1+- control plane节点伸缩
2+ 
3+ 修改有关集群的KubeadmControlPlane.spec.replicas值
4+ 
5+- worker 节点伸缩
6+ 
7+ 修改有关集群的MachineDeployment.spec.replicas值
8+
Adesign/BKE设计文档/image/README.md+1-0
@@ -0,0 +1 @@
1+## BKE Design
Adesign/BKE设计文档/概览/bkeadm.pdf+0-0
Binary files do not support preview
Adesign/BKE设计文档/概览/使用手册.md+143-0
@@ -0,0 +1,143 @@
1+#### 准备环境
2+ 
3+- 首选准备N台裸机，将其划分为引导节点、管理集群、业务集群三种类型，关于集群的划分及要求请参考[环境要求](环境要求.md)
4+ 
5+#### 引导节点
6+ 
7+##### 初始化引导节点
8+ 
9+- 首先将压缩包拷贝到引导节点，然后进行解压缩
10+ 
11+```shell
12+# copy to node
13+$ mkdir /bke
14+$ tar zxvf bke.tar.gz -C /bke
15+```
16+ 
17+- 解压后目录结构如下
18+ 
19+```shell
20+bke/
21+├── bin
22+│   ├── bke
23+│   ├── kind
24+│   └── kubectl
25+└── volumes
26+ ├── docker_registry.tar
27+ ├── nfs.tar
28+ ├── registry.tar
29+ ├── yum_registry.tar
30+ └── yum.tar
31+```
32+ 
33+- 进入/bke目录执行 `./bke init`
34+ 
35+```shell
36+$ cd /bke/bin
37+$ ./bke init
38+# 该指令将检查节点、安装docker、启动kind集群、启动docker_registry yum_registry等
39+# 在/bke/cluster目录下会生成部署集群的模板文件
40+```
41+ 
42+##### 部署管理集群
43+ 
44+- 修改配置文件
45+ 
46+```shell
47+$ ls /bke/cluster
48+cluster.yaml kubeadmconfig.yaml node.json bkecluster.yaml bkemachine.yaml ...
49+# 编辑示例node.json文件，填写真实的节点信息
50+$ cat /bke/cluster/node.json
51+[
52+{"host": "192.168.56.120", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "master-1", "role": "master,etcd"},
53+{"host": "192.168.56.121", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "master-2", "role": "master,etcd"},
54+{"host": "192.168.56.122", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "master-3", "role": "master,etcd"},
55+{"host": "192.168.56.123", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "worker-4", "role": "worker"},
56+{"host": "192.168.56.124", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "worker-5", "role": "worker"}
57+]
58+# 修改bkecluster.yaml 配置addon组件
59+$ vim /bke/cluster/bkecluster.yaml
60+...
61+bkeagent: true # 表示集群部署完成后 bkeagent 接收该集群 apiserver 指令
62+addon
63+- name: calico
64+ version: v3.5.1
65+ param:
66+ - repo: docker.io
67+ env1: env2
68+ - name: boc
69+ verson: v2.2
70+ - name: cluster-api
71+ version: v1.1
72+
73+```
74+ 
75+- 部署管理集群，使用如下命令部署管理集群
76+ 
77+```shell
78+$ bke deploy cluster --cluster=/bke/cluster --kubeconfig=/root/.kube/config
79+# ---- 该命令会持续输出，直到集群部署完成 ---
80+# 该命令的输出数据来源于kubectl get events -l crd=bkecluster
81+# 当集群启动完成后会自动部署BKECLUSTER中定义的addon内容（比如calico/prometheus/carina）
82+...
83+```
84+ 
85+###### 可选
86+ 
87+- （可选）如果未在bkecluster配置addon [boc]，可使用如下指令部署boc
88+ 
89+```shell
90+$ bke deploy product --prod=boc --image=image:v3.5 --kubeconfig=/root/.kube/config
91+```
92+ 
93+- （可选）如果未在管理集群部署cluster-api，可使用如下命令部署
94+ 
95+```shell
96+$ bke deploy product --prod=cluster-api --image=image:v3.5 --kubeconfig=/root/.kube/config
97+```
98+ 
99+- （可选）部署完成管理集群后，kind可以移除
100+ 
101+```shell
102+$ bke remove kind
103+```
104+ 
105+##### 部署业务集群
106+ 
107+- 当管理集群部署完毕后，可以通过boc界面添加业务集群，也可以通过后台添加
108+ 
109+```shell
110+# 生成一个三主三从的集群配置
111+$ bke config cluster -d /bke/bussiness
112+# 编辑示例node.json文件，填写真实的节点信息
113+$ cat /bke/bussiness/node.json
114+[
115+{"host": "192.168.56.120", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "master-1", "role": "master,etcd"},
116+{"host": "192.168.56.121", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "master-2", "role": "master,etcd"},
117+{"host": "192.168.56.122", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "master-3", "role": "master,etcd"},
118+{"host": "192.168.56.123", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "worker-4", "role": "worker"},
119+{"host": "192.168.56.124", "port": "22", "username": "<ssh-user>", "password": "<ssh-password>", "hostname": "worker-5", "role": "worker"}
120+]
121+# 修改bkecluster.yaml 配置addon组件
122+$ car /bke/cluster/bkecluster.yaml
123+$ vim bkecluster.yaml
124+...
125+bkeagent: true # 表示集群部署完成后 bkeagent 接收该集群 apiserver 指令
126+addon
127+- name: calico
128+ version: v3.5.1
129+ param:
130+ - repo: docker.io
131+ env1: env2
132+$ bke deploy cluster --cluster /bke/bussiness --kubeconfig=/root/.kube/config
133+```
134+ 
135+- 部署业务集群
136+ 
137+```shell
138+$ bke deploy cluster --cluster=/bke/bussiness --kubeconfig=/root/.kube/config
139+# ---- 该命令会持续输出，直到集群部署完成 ---
140+# 该命令的输出数据来源于kubectl get events -l crd=bkecluster
141+# 当集群启动完成后会自动部署BKECLUSTER中定义的addon内容（比如calico/prometheus/carina）
142+...
143+```
Adesign/BKE设计文档/概览/架构.md+45-0
@@ -0,0 +1,45 @@
1+#### 初始架构
2+ 
3+> 在裸机状态下完成管理集群（meta）的部署
4+ 
5+①：选择一个低配置裸机节点，作为引导节点，将bke包传至该机器
6+ 
7+②：使用bke启动 镜像仓库 和 kind集群，并在kind集群内部署好cluster-api
8+ 
9+③：选择管理集群（meta），配置好cluster.yaml，并提交到kind集群
10+ 
11+④：cluster-api将bkeagent分发到各个节点并启动，bkeagent会监听kind集群内command crd资源
12+ 
13+⑤：cluster-api发布集群启动指令，bkeagent监听到指令资源创建便开始工作
14+ 
15+⑥：等待集群启动完毕后，cluster-api进行addon组件安装，addon组件是可配置的可以选择自带calico也可按照boc
16+ 
17+⑦：管理集群搭建完成，且在管理集群boc已经部署完成，此时管理集群便失去其作用了，将cluster-api迁移到管理集群，这样便进入了最终架构
18+ 
19+![](../image/cluster-api-arch-1.png)
20+ 
21+ 
22+ 
23+#### 最终架构
24+ 
25+> 当引导节点完成引导集群安装后便失去了作用，目前上边只遗留了Nexus服务，可以选择迁移也可以保留
26+ 
27+①：cluster-api 、 boc等产品已经在该集群启动
28+ 
29+②：支持两种方式部署业务集群，第一种直接在集群提交部署yaml，第二种通过bke(bkeadm)进行业务集群安装
30+ 
31+![](../image/cluster-api-arch-2.png)
32+ 
33+ 
34+ 
35+#### 动态架构
36+ 
37+> 展示bke使用的整个过程
38+ 
39+![](../image/cluster-api.gif)
40+ 
41+#### 时序图
42+ 
43+> 部署时序图
44+ 
45+![](../image/cluster-api-sequence.png)
Adesign/BKE设计文档/概览/环境要求.md+73-0
@@ -0,0 +1,73 @@
1+ 
2+ 
3+#### 硬件要求
4+ 
5+> BKE将机器分为三种类型，引导节点、管理集群、业务集群
6+ 
7+引导节点：用来单独作为安装的节点，不能加入管理集群和业务集群
8+ 
9+管理集群：用来部署BOC产品，至少需要一台8核16G，100G系统盘的机器
10+ 
11+业务集群：业务集群在部署完管理集群后在部署
12+ 
13+##### 最小化部署硬件配置：
14+ 
15+| 节点类型 | 节点数量 | CPU核心 | 内存 | 系统盘 | 数据盘 |
16+| -------------- | -------- | ------- | ---- | ------ | ------ |
17+| 引导节点 | 1 | 1 | 2G | 100G | |
18+| 管理集群 | 1 | 8 | 16G | 100G | |
19+| 业务集群Master | 1 | 8 | 16G | 100G | |
20+| 业务集群Node | 1 | 8 | 16G | 100G | |
21+ 
22+##### 推荐硬件配置
23+ 
24+| 节点类型 | 节点数量 | CPU核心 | 内存 | 系统盘 | 数据盘 |
25+| -------------- | -------- | ------- | ---- | ------ | -------------------------------- |
26+| 引导节点 | 1 | 1 | 2G | 100G | 100G |
27+| 管理集群 | 3 | 8 | 16G | 100G | 300G*2(/var/lib/docker + carina) |
28+| 业务集群Master | 3 | 16 | 32G | 100G | 300G(/var/lib/docker) |
29+| 业务集群Node | 3 | 16 | 32G | 100G | 300G(/var/lib/docker) |
30+ 
31+> 注意：上表中的数据盘（/var/lib/docker）表示的是docker相关信息在主机中的存储位置，即容器数据盘，包括docker的镜像、容器、日志等文件。建议给此路径挂盘避免与系统盘混用，避免因容器、镜像、日志等相关信息导致磁盘压力过大
32+ 
33+#### 软件要求
34+ 
35+| 需求项 | 具体要求 | 参考命令（以centos 7.6为例） |
36+| ---------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
37+| 操作系统 | Ubuntu 16.04/18.04 LTS(64-bit）<br>CentOS Linux 7.6(64-bit) <br> | cat /etc/redhat-release |
38+| kernel版本 | \>= Kernel 3.10.0-957.10.1.el7.x86_64 | uname -sr |
39+| ssh <br> sudo <br> yum | 确保引导节点、管理集群、业务集群能够使用ssh互联 | 1. 确保添加所有节点时，IP、用户名和密码输入正确。<br>2. 确保每个节点都有sudo或root权限。<br>3. 如果是Centos确保拥有yum;其他操作系统类型，确保拥有包管理器<br>4. 确保拥有执行命令工具 |
40+| swap | 关闭 | sudo swapoff -a` `sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab` `# 注意：如果 /etc/fstab有挂载 swap，必须要注释掉，不然重新开机时又会重新挂载 swap |
41+| SELinux | 关闭 | setenforce 0` `sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config |
42+| 时区 | 所有服务器时区必须统一，建议设置为 `Asia/Shanghai` | timedatectl set-timezone Asia/Shanghai |
43+| 时间同步 | ETCD 集群各机器需要时间同步，可以利用 chrony 用于系统时间同步；所有服务器要求时间必须同步，误差不得超过 2 秒 | yum install -y chronyd` `systemctl enable chronyd && systemctl start chronyd |
44+| | | |
45+ 
46+ 
47+ 
48+#### 开放端口
49+ 
50+ 
51+ 
52+| 协议 | 端口 | 节点 | 描述 |
53+| ------- | ---------- | -------------- | ----------------------- |
54+| TCP | 22 | 所有节点 | 首次连接节点使用<br> |
55+| TCP | 6443 | K8S Master节点 | Kubernetes apiserver |
56+| TCP | 2379 | etcd nodes | Etcd client requests |
57+| TCP | 2380 | etcd nodes | etcd peer communication |
58+| TCP | 10250 | 所有节点 | kubelet |
59+| TCP/UDP | 3000-32767 | 所有节点 | NodePort port range |
60+| TCP | 9099 | 管理节点 | calico |
61+| TCP | 9100 | 管理节点 | calico |
62+| TCP | 179 | 管理节点 | calico bgp |
63+| TCP | 53 | 所有节点 | dns |
64+| TCP | 8081 | 引导节点 | nexus |
65+| TCP | 5000 | 引导节点 | nexus 镜像仓库、yum源 |
66+| | | | |
67+| | | | |
68+| | | | |
69+| | | | |
70+| | | | |
71+| | | | |
72+| | | | |
73+ 
Adesign/BKE设计文档/概览/项目简要.md+19-0
@@ -0,0 +1,19 @@
1+#### bkeadm
2+ 
3+- bke是工作为引导节点的二进制文件，它是bke工作的起使，主要功能包括启动镜像仓库、yaml仓库 、kind集群、提交集群配置、部署产品等
4+ 
5+#### cluster-api-provider-metal
6+ 
7+- 该项目是cluster-api的一个插件，其负责翻译cluster-api前置组件生成的部署指令，并将其翻译成bkeagent要执行的指令
8+ 
9+#### bkeagent
10+ 
11+- 定位于云原生的宿主机管理工具，负责监听kube-apiserver指令并在宿主机上执行
12+ 
13+#### cluster-api-provider-controllerplane
14+ 
15+- 迁移官方的cluster-api项目中的controllerplane的实现，修改其生成证书的功能
16+ 
17+#### Manifests
18+ 
19+- 该项目是一个收集诸多部署资源的项目，比如calico beyondac bcc等，这些资源将会被打包成一个镜像，该容器将会作为一个sidecar挂载到cluster-api-provider-metal容器中，这样历次版本发布都需要打包这样的一个镜像
Adesign/BKE设计文档/相关技术/部署参考/containerd安装.md+369-0
@@ -0,0 +1,369 @@
1+## containerd安装
2+ 
3+ 
4+ 
5+1. 部署机器角色
6+ 
7+ - 部署机器, portal集群,业务集群
8+ 
9+2. 部署方式
10+ 
11+ - 二进制部署
12+ 
13+3. 部署前提
14+ 
15+ - yum源启动且可用 参考[yum仓库安装.md](./yum仓库安装.md)
16+ - 本地配置yum源的repo
17+ 
18+4. 部署文件
19+ 
20+ `文档服务器: /Share/products/Kubernetes/konk/rpm_binary/containerd`
21+ 
22+
23+ 
24+ ```
25+ containerd-1.6.0-amd64
26+ containerd-1.6.0-arm64
27+ containerd-shim-1.6.0-amd64
28+ containerd-shim-1.6.0-arm64
29+ containerd-shim-runc-v1-1.6.0-amd64
30+ containerd-shim-runc-v1-1.6.0-arm64
31+ containerd-shim-runc-v2-1.6.0-amd64
32+ containerd-shim-runc-v2-1.6.0-arm64
33+ ctr-1.6.0-amd64
34+ ctr-1.6.0-arm64
35+ crictl-v1.21.0-amd64
36+ crictl-v1.21.0-arm64
37+ nerdctl-0.12.1-amd64
38+ nerdctl-0.12.1-arm64
39+ config.toml # /etc/containerd/config.toml
40+ containerd.service #/usr/lib/systemd/system/containerd.service
41+ crictl.yaml # /etc/crictl.yaml
42+ ```
43+ 
44+5. 部署过程
45+ 
46+ 1. 下载二进制文件到对应位置
47+ 
48+ ```bash
49+ # arm版本替换下载文件名
50+ wget http://deploy.bocloud.k8s:40080/files/containerd-1.6.0-amd64 -O /usr/bin/containerd
51+ wget http://deploy.bocloud.k8s:40080/files/containerd-shim-1.6.0-amd64 -O /usr/bin/containerd-shim
52+ wget http://deploy.bocloud.k8s:40080/files/containerd-shim-runc-v1-1.6.0-amd64 -O /usr/bin/containerd-shim-runc-v1
53+ wget http://deploy.bocloud.k8s:40080/files/containerd-shim-runc-v2-1.6.0-amd64 -O /usr/bin/containerd-shim-runc-v2
54+ wget http://deploy.bocloud.k8s:40080/files/ctr-1.6.0-amd64 -O /usr/bin/ctr
55+ ```
56+ 
57+ 2. crictl工具配置文件
58+ 
59+ ```yaml
60+ runtime-endpoint: unix:///var/run/containerd/containerd.sock
61+ image-endpoint: unix:///var/run/containerd/containerd.sock
62+ debug: false
63+ pull-image-on-create: false
64+ disable-pull-on-run: false
65+ ```
66+ 
67+
68+ 
69+ 3. containerd配置文件config.toml
70+ 
71+ ```toml
72+ disabled_plugins = []
73+ imports = []
74+ oom_score = 0
75+ plugin_dir = ""
76+ required_plugins = []
77+ root = "/var/lib/containerd"
78+ state = "/run/containerd"
79+ version = 2
80+
81+ [cgroup]
82+ path = ""
83+
84+ [debug]
85+ address = ""
86+ format = ""
87+ gid = 0
88+ level = ""
89+ uid = 0
90+
91+ [grpc]
92+ address = "/run/containerd/containerd.sock"
93+ gid = 0
94+ max_recv_message_size = 16777216
95+ max_send_message_size = 16777216
96+ tcp_address = ""
97+ tcp_tls_cert = ""
98+ tcp_tls_key = ""
99+ uid = 0
100+
101+ [metrics]
102+ address = ""
103+ grpc_histogram = false
104+
105+ [plugins]
106+
107+ [plugins."io.containerd.gc.v1.scheduler"]
108+ deletion_threshold = 0
109+ mutation_threshold = 100
110+ pause_threshold = 0.02
111+ schedule_delay = "0s"
112+ startup_delay = "100ms"
113+
114+ [plugins."io.containerd.grpc.v1.cri"]
115+ disable_apparmor = false
116+ disable_cgroup = false
117+ disable_hugetlb_controller = true
118+ disable_proc_mount = false
119+ disable_tcp_service = true
120+ enable_selinux = false
121+ enable_tls_streaming = false
122+ ignore_image_defined_volumes = false
123+ max_concurrent_downloads = 3
124+ max_container_log_line_size = -1
125+ netns_mounts_under_state_dir = false
126+ restrict_oom_score_adj = false
127+ sandbox_image = "{{ kube_pause_image }}:{{ kube_pause_image_tag }}"
128+ selinux_category_range = 1024
129+ stats_collect_period = 10
130+ stream_idle_timeout = "4h0m0s"
131+ stream_server_address = "127.0.0.1"
132+ stream_server_port = "0"
133+ systemd_cgroup = false
134+ tolerate_missing_hugetlb_controller = true
135+ unset_seccomp_profile = ""
136+
137+ [plugins."io.containerd.grpc.v1.cri".cni]
138+ bin_dir = "/opt/cni/bin"
139+ conf_dir = "/etc/cni/net.d"
140+ conf_template = ""
141+ max_conf_num = 1
142+
143+ [plugins."io.containerd.grpc.v1.cri".containerd]
144+ {% if kube_beyondvm_enable | bool %}
145+ default_runtime_name = "richrunc"
146+ {% else %}
147+ default_runtime_name = "runc"
148+ {% endif %}
149+ disable_snapshot_annotations = true
150+ discard_unpacked_layers = false
151+ no_pivot = false
152+ snapshotter = "overlayfs"
153+
154+ [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
155+ base_runtime_spec = ""
156+ container_annotations = []
157+ pod_annotations = []
158+ privileged_without_host_devices = false
159+ runtime_engine = ""
160+ runtime_root = ""
161+ runtime_type = ""
162+
163+ [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
164+
165+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
166+
167+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
168+ base_runtime_spec = ""
169+ container_annotations = []
170+ pod_annotations = []
171+ privileged_without_host_devices = false
172+ runtime_engine = ""
173+ runtime_root = ""
174+ runtime_type = "io.containerd.runc.v2"
175+
176+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
177+ BinaryName = "/usr/local/sbin/runc"
178+ CriuImagePath = ""
179+ CriuPath = ""
180+ CriuWorkPath = ""
181+ IoGid = 0
182+ IoUid = 0
183+ NoNewKeyring = false
184+ NoPivotRoot = false
185+ Root = ""
186+ ShimCgroup = ""
187+ SystemdCgroup = true
188+ [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.richrunc]
189+ base_runtime_spec = ""
190+ container_annotations = []
191+ pod_annotations = []
192+ privileged_without_host_devices = false
193+ runtime_engine = ""
194+ runtime_root = ""
195+ runtime_type = "io.containerd.runc.v2"
196+
197+ [ plugins."io.containerd.grpc.v1.cri".containerd.runtimes.richrunc.options ]
198+ BinaryName = "{{ kube_addon_dir }}/beyondVM/runc_1.1/runc"
199+ CriuImagePath = ""
200+ CriuPath = ""
201+ CriuWorkPath = ""
202+ IoGid = 0
203+ IoUid = 0
204+ NoNewKeyring = false
205+ NoPivotRoot = false
206+ Root = ""
207+ ShimCgroup = ""
208+ SystemdCgroup = true
209+ [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
210+ base_runtime_spec = ""
211+ container_annotations = []
212+ pod_annotations = []
213+ privileged_without_host_devices = false
214+ runtime_engine = ""
215+ runtime_root = ""
216+ runtime_type = ""
217+
218+ [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
219+
220+ [plugins."io.containerd.grpc.v1.cri".image_decryption]
221+ key_model = "node"
222+
223+ [plugins."io.containerd.grpc.v1.cri".registry]
224+ config_path = ""
225+
226+ [plugins."io.containerd.grpc.v1.cri".registry.auths]
227+
228+ [plugins."io.containerd.grpc.v1.cri".registry.configs]
229+ [plugins."io.containerd.grpc.v1.cri".registry.configs."deploy.bocloud.k8s:40443".tls]
230+ insecure_skip_verify = true
231+ [plugins."io.containerd.grpc.v1.cri".registry.headers]
232+
233+ [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
234+ [plugins."io.containerd.grpc.v1.cri".registry.mirrors."deploy.bocloud.k8s:40443"]
235+ endpoint = ["https://deploy.bocloud.k8s:40443"]
236+ [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
237+ tls_cert_file = ""
238+ tls_key_file = ""
239+
240+ [plugins."io.containerd.internal.v1.opt"]
241+ path = "/opt/containerd"
242+
243+ [plugins."io.containerd.internal.v1.restart"]
244+ interval = "10s"
245+
246+ [plugins."io.containerd.metadata.v1.bolt"]
247+ content_sharing_policy = "shared"
248+
249+ [plugins."io.containerd.monitor.v1.cgroups"]
250+ no_prometheus = false
251+
252+ [plugins."io.containerd.runtime.v1.linux"]
253+ no_shim = false
254+ runtime = "runc"
255+ runtime_root = ""
256+ shim = "containerd-shim"
257+ shim_debug = false
258+
259+ [plugins."io.containerd.runtime.v2.task"]
260+ platforms = ["linux/amd64"]
261+
262+ [plugins."io.containerd.service.v1.diff-service"]
263+ default = ["walking"]
264+
265+ [plugins."io.containerd.snapshotter.v1.aufs"]
266+ root_path = ""
267+
268+ [plugins."io.containerd.snapshotter.v1.btrfs"]
269+ root_path = ""
270+
271+ [plugins."io.containerd.snapshotter.v1.devmapper"]
272+ async_remove = false
273+ base_image_size = ""
274+ pool_name = ""
275+ root_path = ""
276+
277+ [plugins."io.containerd.snapshotter.v1.native"]
278+ root_path = ""
279+
280+ [plugins."io.containerd.snapshotter.v1.overlayfs"]
281+ root_path = ""
282+
283+ [plugins."io.containerd.snapshotter.v1.zfs"]
284+ root_path = ""
285+
286+ [proxy_plugins]
287+
288+ [stream_processors]
289+
290+ [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
291+ accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
292+ args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
293+ env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
294+ path = "ctd-decoder"
295+ returns = "application/vnd.oci.image.layer.v1.tar"
296+
297+ [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
298+ accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
299+ args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
300+ env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
301+ path = "ctd-decoder"
302+ returns = "application/vnd.oci.image.layer.v1.tar+gzip"
303+
304+ [timeouts]
305+ "io.containerd.timeout.shim.cleanup" = "5s"
306+ "io.containerd.timeout.shim.load" = "5s"
307+ "io.containerd.timeout.shim.shutdown" = "3s"
308+ "io.containerd.timeout.task.state" = "2s"
309+
310+ [ttrpc]
311+ address = ""
312+ gid = 0
313+ uid = 0
314+ ```
315+ 
316+
317+ 
318+ 4. containerd服务service文件
319+ 
320+ ```ini
321+ # Copyright The containerd Authors.
322+ #
323+ # Licensed under the Apache License, Version 2.0 (the "License");
324+ # you may not use this file except in compliance with the License.
325+ # You may obtain a copy of the License at
326+ #
327+ # http://www.apache.org/licenses/LICENSE-2.0
328+ #
329+ # Unless required by applicable law or agreed to in writing, software
330+ # distributed under the License is distributed on an "AS IS" BASIS,
331+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
332+ # See the License for the specific language governing permissions and
333+ # limitations under the License.
334+
335+ [Unit]
336+ Description=containerd container runtime
337+ Documentation=https://containerd.io
338+ After=network.target local-fs.target
339+
340+ [Service]
341+ ExecStartPre=-/sbin/modprobe overlay
342+ ExecStart=/usr/bin/containerd
343+
344+ Type=notify
345+ Delegate=yes
346+ KillMode=process
347+ Restart=always
348+ RestartSec=5
349+ # Having non-zero Limit*s causes performance problems due to accounting overhead
350+ # in the kernel. We recommend using cgroups to do container-local accounting.
351+ LimitNPROC=infinity
352+ LimitCORE=infinity
353+ LimitNOFILE=infinity
354+ # Comment TasksMax if your systemd version does not supports it.
355+ # Only systemd 226 and above support this version.
356+ TasksMax=infinity
357+ OOMScoreAdjust=-999
358+
359+ [Install]
360+ WantedBy=multi-user.target
361+ ```
362+ 
363+ 5. 启动服务命令
364+ 
365+ ```bash
366+ systemctl start containerd
367+ ```
368+ 
369+
Adesign/BKE设计文档/相关技术/部署参考/docker-registry安装.md+37-0
@@ -0,0 +1,37 @@
1+## 镜像仓库部署
2+ 
3+1. 部署机器角色
4+ - 部署机器, 执行部署portal集群的机器, 可与portal机器重合
5+ 
6+2. 部署时序
7+ - 部署portal集群动作之前
8+ 
9+3. 部署方式
10+ - 容器部署, 使用shell命令 `docker run xxxxx`
11+ 
12+4. 部署文件
13+ 1. 镜像 `bocloud_deploy_registry_k8s:latest` `文档服务器:/Share/products/Kubernetes/konk/images/docker_image_registry`
14+ 2. 挂载文件 `bocloud_current.tar.gz`
15+ 
16+5. 部署过程
17+ 
18+ ```bash
19+ sudo tar xf bocloud_current.tar.gz -C /var/lib/registry/bocloud_deploy_server
20+ docker rm -f bocloud_deploy_registry_k8s;
21+ docker run --name bocloud_deploy_registry_k8s \
22+ --restart=always \
23+ -p 40443:443 \
24+ -v /var/lib/registry/bocloud_deploy_server:/var/lib/registry/docker \
25+ -d bocloud_deploy_registry_k8s:latest
26+ ```
27+ 
28+6. 服务测试
29+ 
30+ ```
31+ docker pull deploy.bocloud.k8s:40443/public/busybox:latest
32+ ```
33+ 
34+ 
35+ 
36+ 
37+ 
Adesign/BKE设计文档/相关技术/部署参考/docker安装.md+83-0
@@ -0,0 +1,83 @@
1+## docker部署
2+ 
3+ 
4+ 
5+ 
6+ 
7+1. 部署机器角色
8+ 
9+ - 部署机器, portal集群,业务集群
10+ 
11+2. 部署方式
12+ 
13+ - 使用yum部署
14+ 
15+3. 部署前提
16+ 
17+ - yum源启动且可用 参考[yum仓库安装.md](./yum仓库安装.md)
18+ - 本地配置yum源的repo
19+ 
20+4. 部署文件
21+ 
22+ ```bash
23+ # yum repo中docker的rpm包
24+ [root@deploy-229 packages]# ll CentOS/7/amd64/docker-ce/
25+ total 199284
26+ -rwxr-xr-x 1 root root 261632 Jun 14 12:37 audit-2.8.5-4.el7.x86_64.rpm
27+ -rwxr-xr-x 1 root root 104408 Jun 14 12:37 audit-libs-2.8.5-4.el7.x86_64.rpm
28+ -rwxr-xr-x 1 root root 78256 Jun 14 12:37 audit-libs-python-2.8.5-4.el7.x86_64.rpm
29+ -rwxr-xr-x 1 root root 302068 Jun 14 12:37 checkpolicy-2.5-8.el7.x86_64.rpm
30+ -rwxr-xr-x 1 root root 30374084 Jun 14 12:37 containerd.io-1.3.7-3.1.el7.x86_64.rpm
31+ -rwxr-xr-x 1 root root 30381608 Jun 14 12:37 containerd.io-1.3.9-3.1.el7.x86_64.rpm
32+ -rwxr-xr-x 1 root root 40816 Jun 14 12:37 container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm
33+ -rwxr-xr-x 1 root root 25268380 Jun 14 12:37 docker-ce-19.03.13-3.el7.x86_64.rpm
34+ -rwxr-xr-x 1 root root 25519432 Jun 14 12:37 docker-ce-19.03.14-3.el7.x86_64.rpm
35+ -rwxr-xr-x 1 root root 40247476 Jun 14 12:37 docker-ce-cli-19.03.13-3.el7.x86_64.rpm
36+ -rwxr-xr-x 1 root root 40247412 Jun 14 12:37 docker-ce-cli-19.03.14-3.el7.x86_64.rpm
37+ -rwxr-xr-x 1 root root 67720 Jun 14 12:37 libcgroup-0.41-21.el7.x86_64.rpm
38+ -rwxr-xr-x 1 root root 57460 Jun 14 12:37 libseccomp-2.3.1-4.el7.x86_64.rpm
39+ -rwxr-xr-x 1 root root 166012 Jun 14 12:37 libselinux-2.5-15.el7.x86_64.rpm
40+ -rwxr-xr-x 1 root root 241168 Jun 14 12:37 libselinux-python-2.5-15.el7.x86_64.rpm
41+ -rwxr-xr-x 1 root root 154876 Jun 14 12:37 libselinux-utils-2.5-15.el7.x86_64.rpm
42+ -rwxr-xr-x 1 root root 154244 Jun 14 12:37 libsemanage-2.5-14.el7.x86_64.rpm
43+ -rwxr-xr-x 1 root root 115284 Jun 14 12:37 libsemanage-python-2.5-14.el7.x86_64.rpm
44+ -rwxr-xr-x 1 root root 304196 Jun 14 12:37 libsepol-2.5-10.el7.x86_64.rpm
45+ -rwxr-xr-x 1 root root 938736 Jun 14 12:37 policycoreutils-2.5-34.el7.x86_64.rpm
46+ -rwxr-xr-x 1 root root 468316 Jun 14 12:37 policycoreutils-python-2.5-34.el7.x86_64.rpm
47+ -rwxr-xr-x 1 root root 32880 Jun 14 12:37 python-IPy-0.75-6.el7.noarch.rpm
48+ -rwxr-xr-x 1 root root 509108 Jun 14 12:37 selinux-policy-3.13.1-268.el7.noarch.rpm
49+ -rwxr-xr-x 1 root root 7335244 Jun 14 12:37 selinux-policy-targeted-3.13.1-268.el7.noarch.rpm
50+ -rwxr-xr-x 1 root root 635184 Jun 14 12:37 setools-libs-3.3.8-4.el7.x86_64.rpm
51+ [root@deploy-229 packages]# ll CentOS/8/amd64/docker-ce/
52+ total 98320
53+ -rwxr-xr-x 1 root root 356576 Jun 14 12:37 checkpolicy-2.9-1.el8.x86_64.rpm
54+ -rwxr-xr-x 1 root root 30388860 Jun 14 12:37 containerd.io-1.3.7-3.1.el8.x86_64.rpm
55+ -rwxr-xr-x 1 root root 48447 Jun 14 12:37 container-selinux-2.124.0-1.module_el8.2.0+305+5e198a41.noarch.rpm
56+ -rwxr-xr-x 1 root root 25228848 Jun 14 12:37 docker-ce-19.03.13-3.el8.x86_64.rpm
57+ -rwxr-xr-x 1 root root 40264132 Jun 14 12:37 docker-ce-cli-19.03.13-3.el8.x86_64.rpm
58+ -rwxr-xr-x 1 root root 71256 Jun 14 12:37 libcgroup-0.41-19.el8.x86_64.rpm
59+ -rwxr-xr-x 1 root root 257108 Jun 14 12:37 policycoreutils-python-utils-2.9-9.el8.noarch.rpm
60+ -rwxr-xr-x 1 root root 88144 Jun 14 12:37 python3-audit-3.0-0.17.20191104git1c2f876.el8.x86_64.rpm
61+ -rwxr-xr-x 1 root root 129868 Jun 14 12:37 python3-libsemanage-2.9-2.el8.x86_64.rpm
62+ -rwxr-xr-x 1 root root 2345228 Jun 14 12:37 python3-policycoreutils-2.9-9.el8.noarch.rpm
63+ -rwxr-xr-x 1 root root 614916 Jun 14 12:37 python3-setools-4.2.2-2.el8.x86_64.rpm
64+ -rwxr-xr-x 1 root root 858488 Jun 14 12:37 tar-1.30-4.el8.x86_64.rpm
65+ [root@deploy-229 packages]# ll Kylin/V10/arm64/docker/
66+ total 36016
67+ -rwxr-xr-x 1 root root 36741508 Jun 14 12:38 docker-engine-18.09.0-202.ky10.aarch64.rpm
68+ -rwxr-xr-x 1 root root 100456 Jun 14 12:38 libcgroup-0.42.2-1.ky10.aarch64.rpm
69+ -rwxr-xr-x 1 root root 32368 Jun 14 12:38 libcgroup-devel-0.42.2-1.ky10.aarch64.rpm
70+
71+
72+ ```
73+ 
74+
75+ 
76+5. 部署命令
77+ 
78+ ```bash
79+ # 系统以及架构不同, 包名称也不相同, 具体包名和yum源挂载文件夹名称一致
80+ yum install docker # yum install docker-ce
81+ ```
82+ 
83+
Adesign/BKE设计文档/相关技术/部署参考/haproxy安装.md+155-0
@@ -0,0 +1,155 @@
1+## haproxy部署
2+ 
3+ 
4+ 
5+1. 部署主机角色
6+ 
7+ - 多master portal或者业务集群的master节点
8+ 
9+2. 部署时序
10+ 
11+ - kubelet 服务启动之后
12+ 
13+3. 部署方式
14+ 
15+ - K8S集群静态pod
16+ 
17+4. 部署文件
18+ 
19+ 1. pod yaml
20+ 
21+ ```yaml
22+ # 参数
23+ haproxy_image: "deploy.bocloud.k8s:40443/haproxy"
24+ haproxy_image_tag: "2.1.4"
25+ apiserver_dest_port: "36443"
26+ haproxy_config_dir: "/etc/haproxy/"
27+ ```
28+ 
29+ ```jinja
30+ apiVersion: v1
31+ kind: Pod
32+ metadata:
33+ name: haproxy
34+ namespace: kube-system
35+ spec:
36+ containers:
37+ - image: {{ haproxy_image }}:{{ haproxy_image_tag }}
38+ name: haproxy
39+ livenessProbe:
40+ failureThreshold: 8
41+ httpGet:
42+ host: localhost
43+ path: /healthz
44+ port: {{ apiserver_dest_port }}
45+ scheme: HTTPS
46+ volumeMounts:
47+ - mountPath: /usr/local/etc/haproxy
48+ name: haproxyconf
49+ - mountPath: /dev/log
50+ name: log
51+ hostNetwork: true
52+ volumes:
53+ - hostPath:
54+ path: {{ haproxy_config_dir }}
55+ name: haproxyconf
56+ - hostPath:
57+ path: /dev/log
58+ name: log
59+ status: {}
60+ ```
61+ 
62+
63+ 
64+ 2. 配置文件
65+ 
66+ ```yaml
67+ #参数
68+ apiserver_dest_port: "36443"
69+ groups['kubernetes_master']: ['192.168.2.221','192.168.2.222']
70+
71+ portal_enable: true
72+ db_enable: true
73+ db_replicas: "2"
74+ DB_PROXY_PORT: "33316"
75+ db_port: "3316"
76+ groups['db']: ['192.168.2.221']
77+ ```
78+
79+
80+
81+ ```jinja2
82+ # /etc/haproxy/haproxy.cfg
83+ #---------------------------------------------------------------------
84+ # Global settings
85+ #---------------------------------------------------------------------
86+ global
87+ # log /dev/log local0 err
88+ # log /dev/log local1 err
89+ # daemon
90+
91+ #---------------------------------------------------------------------
92+ # common defaults that all the 'listen' and 'backend' sections will
93+ # use if not designated in their block
94+ #---------------------------------------------------------------------
95+ defaults
96+ mode tcp
97+ # log global
98+ option dontlognull
99+ option http-server-close
100+ option forwardfor except 127.0.0.0/8
101+ option redispatch
102+ retries 3
103+ option redispatch
104+ timeout http-request 10s
105+ timeout queue 20s
106+ timeout connect 5s
107+ timeout client 45s
108+ timeout server 45s
109+ timeout http-keep-alive 10s
110+ timeout check 10s
111+
112+ #---------------------------------------------------------------------
113+ # apiserver frontend which proxys to the masters
114+ #---------------------------------------------------------------------
115+ frontend apiserver
116+ bind *:{{ apiserver_dest_port }}
117+ mode tcp
118+ option tcplog
119+ default_backend apiserver
120+
121+ #---------------------------------------------------------------------
122+ # roundrobin balancing for apiserver
123+ #---------------------------------------------------------------------
124+ backend apiserver
125+ option httpchk GET /healthz
126+ http-check expect status 200
127+ mode tcp
128+ # option ssl-hello-chk
129+ balance roundrobin
130+ {% for host in groups['kubernetes_master']%}
131+ server {{ host }} {{ host }}:{{ apiserver_src_port }} check check-ssl verify none
132+ {% endfor %}
133+
134+
135+ {% if portal_enable | bool and db_enable | bool and (db_replicas | int) > 1 and DB_PROXY_PORT is defined %}
136+ # leastconn for db
137+ listen mariadb_cluster
138+ bind 0.0.0.0:{{ DB_PROXY_PORT }}
139+ mode tcp
140+ option tcp-check
141+ balance leastconn
142+ timeout client 1800000ms
143+ timeout server 1800000ms
144+ default-server inter 1s fall 3 rise 2
145+ {% for host in groups['db'] %}
146+ server {{ host }} {{ host }}:{{db_port}} check port {{db_port}} on-error mark-down
147+ {% endfor %}
148+ {% endif %}
149+ ```
150+
151+ 
152+5. 部署过程
153+ 
154+ 1. 将yaml文件放置到路径`/etc/kubernetes/manifests/`
155+ 2. kubelet服务间隔时间拉起静态pod
Adesign/BKE设计文档/相关技术/部署参考/keepalived安装.md+240-0
@@ -0,0 +1,240 @@
1+## keepalied服务部署
2+ 
3+ 
4+ 
5+1. 部署机器角色
6+ - 多master portal或者业务集群的master节点
7+ 
8+2. 部署时序
9+ 
10+ - kubelet 服务启动之后
11+ 
12+3. 部署方式
13+ 
14+ - K8S集群静态pod
15+ 
16+4. 部署文件
17+ 
18+ 1. pod yaml
19+ 
20+ ```yaml
21+ # 文件
22+ /etc/kubernetes/manifests/keepalived.yaml
23+ # 参数
24+ keepalived_image: "deploy.bocloud.k8s:40443/keepalived"
25+ keepalived_image_tag: "1.3.5"
26+ keepalived_config_dir: "/etc/keepalived"
27+ ```
28+ 
29+ ```jinja2
30+ ---
31+ apiVersion: v1
32+ kind: Pod
33+ metadata:
34+ creationTimestamp: null
35+ name: keepalived
36+ namespace: kube-system
37+ spec:
38+ containers:
39+ - image: {{ keepalived_image }}:{{ keepalived_image_tag }}
40+ imagePullPolicy: IfNotPresent
41+ name: keepalived
42+ args:
43+ - --dont-fork
44+ - --log-console
45+ - --log-detail
46+ - --dump-conf
47+ - --use-file=/usr/local/etc/keepalived/keepalived.conf
48+ resources: {}
49+ securityContext:
50+ capabilities:
51+ add:
52+ - NET_ADMIN
53+ - NET_BROADCAST
54+ - NET_RAW
55+ volumeMounts:
56+ - mountPath: /usr/local/etc/keepalived/
57+ name: config
58+ - mountPath: /lib/modules
59+ name: lib-modules
60+ hostNetwork: true
61+ volumes:
62+ - hostPath:
63+ path: {{ keepalived_config_dir }}
64+ name: config
65+ - hostPath:
66+ path: /lib/modules
67+ name: lib-modules
68+ status: {}
69+ ```
70+ 
71+ 2. 配置文件
72+ 
73+ ```yaml
74+ # 文件
75+ /etc/keepalived/keepalived.conf
76+ # 参数
77+ keepalived_config_map: {'role':role,
78+ 'interface': interface,
79+ 'keepalived_virtual_router_id':keepalived_virtual_router_id,
80+ 'keepalived_advert_int':keepalived_advert_int,
81+ 'keepalived_auth_pass':keepalived_auth_pass,
82+ 'vip':vip}
83+ role: "master" # ingress
84+ interface: "ens192"
85+ keepalived_virtual_router_id: "23" #随机数值
86+ keepalived_advert_int: "1"
87+ keepalived_auth_pass: "adIUADkas5" #随机密码
88+ vip: "192.168.2.224"
89+
90+ groups['kubernetes_master']: ['192.168.2.221','192.168.2.222']
91+ groups['ingress']: ['192.168.2.223','192.168.2.224']
92+ ```
93+ 
94+
95+ 
96+ ```jinja2
97+ ! Configuration File for keepalived
98+
99+ global_defs {
100+ router_id LVS_DEVEL
101+ vrrp_skip_check_adv_addr
102+ ! vrrp_strict
103+ vrrp_garp_interval 0
104+ vrrp_gna_interval 0
105+ }
106+
107+ {% for config in keepalived_config_map %}
108+ {% if config['role'] == "master" %}
109+ vrrp_script chk_apiserver {
110+ script "/usr/local/etc/keepalived/check_master.sh"
111+ interval 2
112+ weight -{{ (groups['kubernetes_master'] | length) * 10 }}
113+ }
114+
115+ vrrp_instance VI_1 {
116+ {% if inventory_hostname == groups['kubernetes_master'][0] %}
117+ state MASTER
118+ priority 100
119+ {% else -%}
120+ {% for node in groups['kubernetes_master'] %}
121+ {% if inventory_hostname == node %}
122+ state BACKUP
123+ priority {{ 110 - 10 * loop.index }}
124+ nopreempt
125+ {% endif %}
126+ {% endfor %}
127+ {% endif %}
128+ interface {{ config['interface'] }}
129+ virtual_router_id {{ config['keepalived_virtual_router_id'] }}
130+ advert_int {{config['keepalived_advert_int']}}
131+ authentication {
132+ auth_type PASS
133+ auth_pass {{ config['keepalived_auth_pass'] }}
134+ }
135+ virtual_ipaddress {
136+ {{config['vip']}}
137+ }
138+ track_script {
139+ chk_apiserver
140+ }
141+
142+ }
143+ {% elif config['role'] == "ingress" -%}
144+
145+ vrrp_script check_ingress {
146+ script "/usr/local/etc/keepalived/check_ingress.sh"
147+ weight -{{ (groups['ingress'] | length) * 10 }}
148+ }
149+
150+ vrrp_instance VI_2 {
151+ {% if inventory_hostname == groups['ingress'][0] %}
152+ state MASTER
153+ priority 100
154+ {% else -%}
155+ {% for node in groups['ingress'] %}
156+ {% if inventory_hostname == node %}
157+ state BACKUP
158+ priority {{ 110 - 10 * loop.index }}
159+ nopreempt
160+ {% endif %}
161+ {% endfor %}
162+ {% endif %}
163+ interface {{ config['interface'] }}
164+ virtual_router_id {{ config['keepalived_virtual_router_id'] }}
165+ advert_int {{config['keepalived_advert_int']}}
166+ authentication {
167+ auth_type PASS
168+ auth_pass {{ config['keepalived_auth_pass'] }}
169+ }
170+ virtual_ipaddress {
171+ {{config['vip']}}
172+ }
173+ track_script {
174+ check_ingress
175+ }
176+ }
177+ {% endif -%}
178+ {% endfor %}
179+ ```
180+ 
181+ 3. 脚本文件
182+ 
183+ ```yaml
184+ # 文件
185+ /etc/keepalived/check_master.sh
186+ # 参数
187+ apiserver_dest_port: "36443"
188+ master_vip: "192.168.2.224"
189+ ```
190+
191+
192+
193+ ```jinja2
194+ #!/bin/sh
195+ errorExit() {
196+ echo "*** $*" 1>&2
197+ exit 1
198+ }
199+
200+ curl --silent --max-time 2 --insecure https://localhost:{{ apiserver_dest_port }}/ -o /dev/null || errorExit "Error GET https://localhost:{{ apiserver_dest_port}}/"
201+ if ip addr | grep -q {{ master_vip }}; then
202+ curl --silent --max-time 2 --insecure https://{{ master_vip }}:{{ apiserver_dest_port }}/ -o /dev/null || errorExit "Error GET https://{{ master_vip }}:{{ apiserver_dest_port }}/"
203+ fi
204+ ```
205+
206+
207+ ```yaml
208+ # 文件
209+ /etc/keepalived/check_ingress.sh
210+ # 参数
211+ ingress_controller_health_port: "10254"
212+ ingress_mgr_health_port: "9014"
213+ vip: "192.168.2.224"
214+ ```
215+
216+
217+
218+
219+ ```jinja2
220+ #!/bin/sh
221+ errorExit() {
222+ echo "*** $*" 1>&2
223+ exit 1
224+ }
225+
226+ curl --silent --max-time 2 --connect-timeout 10 -s http://localhost:{{ ingress_controller_health_port }}/healthz || errorExit "Error GET https://localhost:{{ ingress_controller_health_port }}/healthz"
227+ curl --silent --max-time 2 --connect-timeout 10 -s http://localhost:{{ ingress_mgr_health_port }}/healthz || errorExit "Error GET https://localhost:{{ ingress_mgr_health_port }}/health"
228+
229+ if ip addr | grep -q {{ vip }}; then
230+ curl --silent --max-time 2 --connect-timeout 10 -s http://localhost:{{ ingress_controller_health_port }}/healthz || errorExit "Error GET https://localhost:{{ ingress_controller_health_port }}/healthz"
231+ curl --silent --max-time 2 --connect-timeout 10 -s http://localhost:{{ ingress_mgr_health_port }}/healthz || errorExit "Error GET https://localhost:{{ ingress_mgr_health_port }}/health"
232+ fi
233+ ```
234+
235+
236+ 
237+5. 部署过程
238+ 
239+ 1. 将yaml文件放置到路径`/etc/kubernetes/manifests/`
240+ 2. kubelet服务间隔时间拉起静态pod
Adesign/BKE设计文档/相关技术/部署参考/yum仓库安装.md+141-0
@@ -0,0 +1,141 @@
1+## yum仓库部署
2+ 
3+ 
4+ 
5+1. 部署机器角色
6+ 
7+ - 部署机器, 执行部署portal集群的机器, 可与portal机器重合
8+ 
9+2. 部署时序
10+ 
11+ - 部署portal集群动作之前
12+ 
13+3. 部署方式
14+ 
15+ - 容器部署, 使用shell命令 `docker run xxxxx`
16+ 
17+4. 部署文件
18+ 
19+ - 镜像: `deploy.bocloud.k8s:40443/yum_registry:latest`
20+ 
21+ ```
22+ # 镜像文件
23+ yum_registry_amd64.tar
24+ yum_registry_arm64.tar
25+ ```
26+ 
27+ - 挂载文件: `packages/`
28+ 
29+ ```
30+ [root@deploy-229 docker]# tree packages -d
31+ packages
32+ ├── CentOS # 支持的系统名称
33+ │   ├── 7 # 系统版本
34+ │   │   ├── amd64 # 系统架构, yum repo文件中路径 http:/xxxx:xxx/CentOS/7/amd64
35+ │   │   │   ├── ansible # yum install ansible
36+ │   │   │   ├── docker-ce # yum install docker-ce
37+ │   │   │   ├── k8s # yum install kubectl-1.17.11
38+ │   │   │   ├── lxcfs
39+ │   │   │   ├── mariadb
40+ │   │   │   ├── nfs-utils
41+ │   │   │   ├── ntp
42+ │   │   │   ├── openvswitch
43+ │   │   │   ├── repodata # repo元数据
44+ │   │   │   └── tools # 放置一些单独的工具的rpm包 yum install jq
45+ │   │   └── arm64
46+ │   └── 8
47+ │   ├── amd64
48+ │   │   ├── ansible
49+ │   │   ├── docker-ce
50+ │   │   ├── k8s
51+ │   │   ├── lxcfs
52+ │   │   ├── mariadb-client
53+ │   │   ├── nfs-utils
54+ │   │   ├── openvswitch
55+ │   │   ├── repodata # repo元数据
56+ │   │   └── tools
57+ │   └── arm64
58+ ├── files # files中文件可以使用wget/curl http协议下载
59+ └── Kylin
60+ └── V10
61+ ├── amd64
62+ └── arm64
63+ ├── ansible
64+ ├── docker
65+ ├── k8s
66+ ├── lxcfs
67+ │   └── lxcfs
68+ ├── mariadb
69+ ├── nfs-utils
70+ ├── others
71+ ├── pip3
72+ ├── repodata # repo元数据
73+ └── tools
74+ ```
75+ 
76+
77+ 
78+5. 部署过程
79+ 
80+ ```bash
81+ # 解压挂载文件
82+ tar zxvf ./packages.tar.gz
83+ # 执行shell
84+ docker rm -f yum_registry;
85+ sudo chmod 755 -R ./packages/ &&
86+ docker run --name yum_registry \
87+ --restart=always \
88+ -p {YUM_PORT}:80 \
89+ -v {LOCAL_VOLUME}:{REMOTE_VOLUME} \
90+ -d deploy.bocloud.k8s:40443/yum_registry:latest
91+ ```
92+ 
93+
94+ 
95+6. 服务测试
96+ 
97+ 1. 测试下载文件
98+ 
99+ ```bash
100+ # 下载packages/files/ 下的文件
101+ # 在/etc/hosts中添加yum源主机域名, 替换127.0.0.1为域名
102+ [root@deploy-229 docker]# wget http://127.0.0.1:40080/files/yq-4.9.5-arm64
103+ ```
104+
105+
106+
107+ 2. 测试yum源
108+
109+ ```bash
110+ #1. yum使用本地yum源
111+ [root@deploy-229 7]# cat /etc/yum.repos.d/local_uboc.repo
112+ [local_uboc]
113+ baseurl = file:///home/boc3.6/paas_auto_deploy_visualization_v0.1/packages/CentOS/8/amd64
114+ enabled = 1
115+ gpgcheck = 0
116+ name = yum repo
117+
118+ #2. yum使用远程yum源, 可在/etc/hosts中添加yum源主机域名, 替换127.0.0.1为域名
119+ [root@deploy-223 ~]# cat /etc/yum.repos.d/uboc.repo
120+ [uboc]
121+ baseurl = http://192.168.2.229:40080//CentOS/7/amd64
122+ enabled = 1
123+ gpgcheck = 0
124+ name = yum repo
125+
126+ # 查看自建源
127+ [root@deploy-223 ~]# yum repolist
128+ #下载jq rpm包到本地
129+ [root@deploy-229 docker]# yum install jq --downloadonly --downloaddir=./ -y
130+ ```
131+
132+ 3. 手动更新repo元数据
133+
134+ ```bash
135+ yum install -y createrepo
136+ cd packages/CentOS/7/
137+ createrepo amd64/
138+ ```
139+
140+
141+