3 主 3 从高可用 K8S 集群部署

一、在部署节点设置集群节点hosts

cat << EOF >> /etc/hosts
172.16.100.9 master01
172.16.100.10 master02
172.16.100.11 master03
172.16.100.12 worker01
172.16.100.13 worker02
172.16.100.14 worker03
EOF

二、设置配置文件

从 github 拉取项目

git clone git@github.com:pixiu-io/kubez-ansible.git

修改 etc/kuubez/global.yaml

#####################
# kubernetes options
#####################
# Enable an high availability kubernetes cluster.
enable_kubernetes_ha: "yes"

kube_release: 1.32.4

cluster_cidr: "172.30.0.0/16"
service_cidr: "10.254.0.0/16"

#Network interface is optional, the default value
#is eth0.
network_interface: "eth0"

# This should be a VIP, an unused IP on your network that will float between
# the hosts running keepalived for high-availability.
kube_vip_address: "172.16.100.100"

# Enable haproxy and keepalived
# This configuration is usually enabled when self-created VMs require high availability.
enable_haproxy: "yes"

# Listen port for kubernetes.
# 启用 haproxy + keepalived 时, 监听端口推荐使用 8443
kube_vip_port: 8443

# Kubernetes network cni options
enable_calico: "yes"

# kubernetes 镜像仓库地址，默认阿里云，用户可根据实际情况配置
# 可使用 pixiu 社区镜像仓库：docker.io/pixiuio
image_repository: "registry.cn-hangzhou.aliyuncs.com/google_containers"

# 自定义镜像仓库
app_image_repository: "ccr.ccs.tencentyun.com/pixiucloud"

# 自定义 REPO 仓库
repo_dir: "/etc/kubez/repo"

###############
# Host Options
###############
# 自动重置节点的 hostname
# Rocky 系统不生效
set_hostname: "yes"

##################
# Runtime Options
##################
# runtime docker version
# 通过 apt-cache madison docker-ce 查找合适版本
docker_release: ""
# runtime containerd version
containerd_release: ""

# 自定义 docker 数据存放目录
docker_data_dir: "/data/docker"
# 自定义 containerd 数据存放目录
containerd_data_dir: "/data/containerd"

####################
# kubernetes(1.24.0+) 使用 runtime 为 docker
####################
# cir-dockerd 的二进制文件镜像
#cri_dockerd_image: "ccr.ccs.tencentyun.com/pixiucloud/cri-dockerd:v0.3.10"

#####################
# keepalived options
#####################
# Arbitrary unique number from 0..255
keepalived_virtual_router_id: "51"
...
###############
# Helm Options
###############
enable_helm: "yes"

# v3.5.2 and v3.9.0 is available, default is v3.9.0
helm_release: v3.9.0
helm_image: "ccr.ccs.tencentyun.com/pixiucloud/helm-toolbox:{{ helm_release }}"
...
#######################
# Ingress Nginx Options
#######################
enable_ingress_nginx: "yes"

如果服务器是云服务器，需要先创建一个虚拟 IP，然后把这个虚拟 IP 绑定到 master 节点上
修改 ansible/inventory/multinode

# When master nodes and woker nodes use different interfaces,
# you need to comment out "api_interface" from the globals.yml
# and specify like below:
#kube01  api_interface=ep66

# User local connection in the assign groups, such as:
#[docker-master]
#localhost       ansible_connection=local

# These hostname must be resolvable from your deployment host
# The above can also be specified as follows:
# kube[01:02]

[docker-master]

[docker-node]


[containerd-master]
master01
master02
master03

[containerd-node]
worker01
worker02
worker03

[storage]
master01

# Don't change the bellow groups
[kube-master:children]
docker-master
containerd-master

[kube-node:children]
docker-node
containerd-node

[baremetal:children]
kube-master
kube-node
storage

[kubernetes:children]
kube-master
kube-node

[nfs-server:children]
storage

[haproxy:children]
kube-master

三、部署k8s集群

复制kubez-ansible到/tmp目录下
拷贝初始化文件到当前目录，并初始化

cp /tmp/kubez-ansible/tools/setup_env.sh ./setup_env.sh
bash setup_env.sh

设置免密登录

ssh-keygen
apt-get install sshpass
kubez-ansible -i multinode  authorized-key

安装k8s需要的依赖

kubez-ansible -i multinode bootstrap-servers

安装k8s集群

kubez-ansible -i multinode deploy

验证集群安装状态

kubectl get nodes

四、安装kubesphere代替原来的 dashboard

helm upgrade --install -n kubesphere-system --create-namespace ks-core \
	https://charts.kubesphere.com.cn/main/ks-core-1.1.4.tgz \
	--set portal.hostname=dashboard.test.com \
	--set portal.https.port=30880 \
	--set ingress.enabled=true \
	--set ingress.tls.source=importation \
	--set ingress.tls.secretName=your_tls_cert \
	--set ingress.ingressClassName=nginx \
	--set global.imageRegistry=swr.cn-southwest-2.myhuaweicloud.com/ks \
	--set extension.imageRegistry=swr.cn-southwest-2.myhuaweicloud.com/ks

# 初始用户名和密码
admin
P@88w0rd