基于Kubernetes v1.25 (二进制) 部署高可用集群

基于k8s 1.25（二进制）部署高可用集群

Kubernetes（简称为：k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。它支持一系列容器工具, 包括Docker、Containerd等。

1.安装说明

本文将演示二进制方式安装高可用k8s 1.25，k8s 1.24+相对于其他版本，二进制安装方式并无太大区别，只需要区分每个组件版本的对应关系即可。生产环境中，建议使用小版本大于5的Kubernetes版本，比如1.24.5以后的才可用于生产环境。

在这里插入图片描述

版本说明：https://kubernetes.io/zh-cn/releases/

2.环境说明

2.1 主机规划

主机地址	主机名	主机配置	主机角色	软件列表
192.168.3.111	k8s-master1	2C4G	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc
192.168.3.112	k8s-master2	2C4G	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc
192.168.3.113	k8s-master3	2C4G	master	kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、Containerd、runc
192.168.3.114	k8s-worker1	2C4G	worker	kubelet、kube-proxy、Containerd、runc
192.168.3.11	ha1	1C2G	LB	haproxy、keepalived
192.168.3.12	ha2	1C2G	LB	haproxy、keepalived
192.168.3.100		VIP(虚拟IP)

2.2 软件版本

软件名称	软件版本	备注
支持的操作系统版本	CentOS 7.9/stream 8、Rocky 8、Ubuntu 18.04/20.04	CentOS7.9内核kernel版本：5.18.0-1.el8
kubernetes	v1.25.0
etcd	v3.5.4	最新版本
containerd	1.6.9
runc	1.1.1
cfssl	v1.6.1
docker	Docker CE 20.10.20	使用最新版本
cri-dockerd	v0.2.5	使用最新版本
calico	v3.19.4
coredns	v1.8.4
haproxy	5.18	YUM源默认
keepalived	3.5	YUM源默认

2.3 网络分配

集群安装时会涉及到三个网段：

宿主机网段：就是安装k8s的服务器

Pod网段：k8s Pod的网段，相当于容器的IP

Service网段：k8s service网段，service用于集群容器通信。

网络名称	网段	备注
Node网络	192.168.3.0/24
Service网络	10.96.0.0/16
Pod网络	10.244.0.0/16

3.环境配置

3.1 设置IP地址

各节点设置ip地址格式如下：

# CentOS
vi /etc/sysconfig/network-scripts/ifcfg-ensXXX
[root@k8s-master1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 
DEVICE=ens33
NAME=ens33
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.3.111
PREFIX=24
GATEWAY=192.168.3.1
DNS1=223.5.5.5
DNS2=180.76.76.76

# Ubuntu
root@k8s-master1:~# cat /etc/netplan/01-netcfg.yaml 
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses: [192.168.3.111/24] 
      gateway4: 192.168.3.1
      nameservers:
        addresses: [223.5.5.5, 180.76.76.76]

3.2 设置主机名

各节点设置主机名：

hostnamectl set-hostname xxx

注：关于主机名参见2.1小节主机规划表

3.3 配置IP和主机名解析

cat >> /etc/hosts << EOF
192.168.3.111 k8s-master1
192.168.3.112 k8s-master2
192.168.3.113 k8s-master3
192.168.3.114 k8s-worker1
192.168.3.11  ha1
192.168.3.11  ha2
EOF

3.3 配置镜像源

cat > /etc/yum.repos.d/centos-tuna.repo << 'EOF'
[base]
name=CentOS-\$releasever - Base
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/os/\$basearch/
enabled=1
gpgcheck=0

#released updates
[updates]
name=CentOS-\$releasever - Updates
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/updates/\$basearch/
#baseurl=https://mirrors.aliyun.com/centos/\$releasever/updates/\$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=\$releasever&arch=\$basearch&repo=updates
enabled=1
gpgcheck=0

#additional packages that may be useful

[centosplus]
name=CentOS-\$releasever - Plus
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/centosplus/\$basearch/
#baseurl=https://mirrors.aliyun.com/centos/\$releasever/centosplus/\$basearch/
#mirrorlist=http://mirrorlist.centos.org/?release=\$releasever&arch=\$basearch&repo=centosplus
enabled=1
gpgcheck=0

[cloud]
name=CentOS-\$releasever - Cloud
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/cloud/\$basearch/openstack-train/
#baseurl=https://mirrors.aliyun.com/centos/\$releasever/cloud/\$basearch/openstack-train/
enabled=1
gpgcheck=0

[paas]
name=CentOS-\$releasever - paas
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/paas/\$basearch/openshift-origin13/
#baseurl=https://mirrors.aliyun.com/centos/\$releasever/paas/\$basearch/openshift-origin13/
enabled=1
gpgcheck=0

[kvm]
name=CentOS-\$releasever - kvm
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/virt/\$basearch/kvm-common/
#baseurl=https://mirrors.aliyun.com/centos/\$releasever/virt/\$basearch/kvm-common/
enabled=1
gpgcheck=0

[extras]
name=CentOS-\$releasever - extras
baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos/\$releasever/extras/\$basearch/
#baseurl=https://mirrors.aliyun.com/centos/\$releasever/extras/\$basearch/
enabled=1
gpgcheck=0

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/$basearch/debug
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=https://mirrors.tuna.tsinghua.edu.cn/epel/7/SRPMS
#mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-7&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0

[elrepo]
name=ELRepo.org Community Enterprise Linux Repository - el7
baseurl=http://mirrors.tuna.tsinghua.edu.cn/elrepo/elrepo/el7/x86_64/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

[elrepo-kernel]
name=ELRepo.org Community Enterprise Linux Kernel Repository - el7
baseurl=http://mirrors.tuna.tsinghua.edu.cn/elrepo/kernel/el7/x86_64/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

[elrepo-extras]
name=ELRepo.org Community Enterprise Linux Extras Repository - el7
baseurl=http://mirrors.tuna.tsinghua.edu.cn/elrepo/extras/el7/x86_64/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-elrepo.org
protect=0

EOF

注：根据网络环境选择阿里云或清华大学源

3.4 配置主机安全

#关闭防火墙
systemctl disable --now firewalld
systemctl disable --now NetworkManager
firewall-cmd --state
#关闭selinux
setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
sestatus

3.5 配置交换分区

swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p

3.6 配置时间同步

#推荐使用chrony配置
yum install -y chrony
#替换为国内时间服务器
cp /etc/chrony.conf /etc/chrony.conf.bak
sed -i -e '/^pool.*/d' -e '/^server.*/d' -e '/^# Please consider .*/a\server ntp.aliyun.com iburst\nserver time1.cloud.tencent.com iburst\nserver ntp.tuna.tsinghua.edu.cn iburst' -e 's@^#allow.*@allow 0.0.0.0/0@' -e 's@^#local.*@local stratum 10@' /etc/chrony.conf

systemctl enable --now chronyd 
chronyc sources
chronyc sources -nv

3.7 优化资源限制

#limit优化
ulimit -SHn 65535

cat >>/etc/security/limits.conf << EOF
* soft nofile 65536
* hard nofile 131072
* soft nproc 65536
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

3.8 系统内核升级

CentOS7内核是3.10，kubernetes需要的内核是4.18或以上版本，CentOS7需要升级内核才可以稳定运行kubernetes各组件，其它系统根据自己的需求去升级。CentOS7 需要升级内核至4.18+，本此升级的版本为5.19。

#安装依赖包
yum -y install perl
#升级内核需要先导入elrepo的key，然后安装elrepo的yum源，启用 ELRepo 仓库
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
#仓库启用后，你可以使用下面的命令列出可用的内核相关包
yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
# -ml	最新版本的版本
# -lt	长期支持的版本
# 也可通过官方源地址查看： http://elrepo.reloumirrors.net/kernel/el7/x86_64/RPMS/

#安装最新的主线稳定内核
yum --enablerepo="elrepo-kernel" -y install kernel-ml kernel-ml-devel.x86_64 

#查看系统可用内核，并设置启动项
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
#指定开机GRUB默认启动内核版本
grub2-set-default 0
#新创建内核配置
grub2-mkconfig -o /etc/grub2.cfg
#检查默认内核是不是5.19
grubby --default-kernel

#重启后再次检查
reboot

uname -r
#注意：原来使用的内核版本建议不要删除，如果遇到问题可以进行回退。

3.9 系统内核优化

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 131072
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF

sysctl --system

#所有节点配置完内核后，重启服务器，保证重启后内核依旧加载
reboot -h now

#重启后查看ipvs模块加载情况：
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

#重启后查看containerd相关模块加载情况：
lsmod | egrep 'br_netfilter|overlay'

3.10 ipvs管理工具安装及模块加载

为集群节点安装，负载均衡节点不用安装

#安装依赖包
yum -y install ipvsadm ipset sysstat conntrack libseccomp

#所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4 已经改为 nf_conntrack，4.18以下使用nf_conntrack_ipv4 即可： 
modprobe -- ip_vs 
modprobe -- ip_vs_rr 
modprobe -- ip_vs_wrr 
modprobe -- ip_vs_sh 
modprobe -- nf_conntrack 

#创建 /etc/modules-load.d/ipvs.conf 并加入以下内容： 
cat >/etc/modules-load.d/ipvs.conf <<EOF 
ip_vs 
ip_vs_lc 
ip_vs_wlc 
ip_vs_rr 
ip_vs_wrr 
ip_vs_lblc 
ip_vs_lblcr 
ip_vs_dh 
ip_vs_sh 
ip_vs_fo 
ip_vs_nq 
ip_vs_sed 
ip_vs_ftp 
ip_vs_sh 
nf_conntrack 
ip_tables 
ip_set 
xt_set 
ipt_set 
ipt_rpfilter 
ipt_REJECT 
ipip 
EOF

#查看ipvs转发
ipvsadm -ln

3.11 加载containerd相关内核模块

#临时加载模块：
modprobe overlay
modprobe br_netfilter

#永久性加载模块：
cat > /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

#设置为开机启动：
systemctl enable --now systemd-modules-load.service

3.11 配置 ssh key免密验证

配置 ssh key 免密验证，方便后续同步文件

#在k8s-master1上操作
ssh-keygen

ssh-copy-id root@k8s-master1
ssh-copy-id root@k8s-master2
ssh-copy-id root@k8s-master3
ssh-copy-id root@k8s-worker1
#验证
ssh root@k8s-master1

#或者通过如下脚本进行配置：
for i in k8s-master1 k8s-master2 k8s-master3 k8s-worker1 ;
do ssh-copy-id -i ~/.ssh/id_rsa.pub $i;
done

3.12 其它工具安装(选装)

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y
yum install -y nfs-utils gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo libaio-devel ncurses-devel autoconf automake zlib-devel python-devel conntrack

4.高可用组件安装

4.1 安装haproxy与keepalived

yum -y install haproxy keepalived

4.2 HAProxy配置

cat > /etc/haproxy/haproxy.cfg << "EOF"
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  k8s-master1  192.168.3.111:6443 check
 server  k8s-master2  192.168.3.112:6443 check
 server  k8s-master3  192.168.3.113:6443 check
EOF

4.3 KeepAlived配置

主从配置不一致，需要注意。
ha1:

cat > /etc/keepalived/keepalived.conf << "EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state MASTER
   interface ens33
   mcast_src_ip 192.168.3.111
   virtual_router_id 51
   priority 100
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       192.168.3.100
   }
   track_script {
      chk_apiserver
   }
}
EOF

ha2:

cat > /etc/keepalived/keepalived.conf << "EOF"
! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL
script_user root
   enable_script_security
}
vrrp_script chk_apiserver {
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1
}
vrrp_instance VI_1 {
   state BACKUP
   interface ens33
   mcast_src_ip 192.168.3.112
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication {
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   }
   virtual_ipaddress {
       192.168.3.100
   }
   track_script {
      chk_apiserver
   }
}
EOF

4.4 健康检查脚本

ha1及ha2均要配置

cat > /etc/keepalived/check_apiserver.sh << "EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi
EOF

chmod +x /etc/keepalived/check_apiserver.sh

4.5 启动服务并验证

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

ip address show
master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy
master节点查看监听端口：netstat -lntp

5. 安装Docker作为Container Runtime

由于我们现在生产环境中有很多环境需要用到docker，所有我们这里也需要安装docker作为容器运行时。

值得注意的是v1.24 的 Kubernetes 正式移除对Dockershim的支持，即默认不再支持 docker。因为1.24的时候k8s已经在江湖上赫赫有名，已经不在依附于docker，当时的容器技术也不仅仅只有docker可选，还有其它的容器可选，因此它就终于独立了，它决定把dockershim这个组件不在开发把它移除，这时候就变成了什么样子了，dockershim就彻底的没有了，1.24把dockershim抛弃了docker就没办法通过CRI和k8s进行通讯，自然k8s也就没有办法来使用docker的容器技术，当然docker也没有办法通过k8s来进行结合使用，这个对于k8s来讲已经不是问题，因为k8s除了docker以外它还有其它的容器技术可选，而docker对应的企业级的编排的领域只有k8s可选，因为k8s已经是垄断地位它没有第三方可选，在这种情况下docker放低身段只好自己开发一个软件贴在CRI基础之上这就是cri-dockerd，cri-dockerd是docker公司开发出来的，人家不兼容了抛弃你了，那我就自己做个软件叫cri-dockerd来遵守CRI从而才能和k8s组合在一起。

在这里插入图片描述

5.1. 安装 Docker

master和worker节点均需要安装docker-ce

#查看支持哪些docker版本
yum list docker-ce --showduplicates |sort -r

# 安装docker
yum remove docker*
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce  docker-ce-cli
# 修改docker配置,新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["http://hub-mirror.c.163.com"]
}
EOF
#或者是如下比较详细的配置
cat > /etc/docker/daemon.json <<-EOF
{
    "registry-mirrors": [
        "https://docker.mirrors.ustc.edu.cn",
        "https://hub-mirror.c.163.com",
        "https://reg-mirror.qiniu.com",
        "https://registry.docker-cn.com"
],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
    	"max-size": "300m",
		"max-file": "2"	
},
    "storage-driver": "overlay2"
    "live-restore": true
}
EOF

systemctl daemon-reload && systemctl enable --now docker

#也可以自己下载rpm离线包进行安装
http://mirrors.aliyun.com/docker-ce/linux/centos/7.9/x86_64/stable/Packages/
yum localinstall xxxx

5.2. 安装 cri-dockerd

Kubernetes自v1.24移除了对docker-shim的支持，而Docker Engine默认又不支持CRI规范，因而二者将无法直接完成整合。为此，Mirantis和Docker联合创建了cri-dockerd项目，用于为Docker Engine提供一个能够支持到CRI规范的垫片，从而能够让Kubernetes基于CRI控制Docker 。

项目地址：https://github.com/Mirantis/cri-dockerd

cri-dockerd项目提供了预制的二制格式的程序包，用户按需下载相应的系统和对应平台的版本即可完成安装。

5.2.1 镜像包方式安装cri-dockerd

到下面的链接下载最新版cri-docker:
https://github.com/Mirantis/cri-dockerd/releases

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.5/cri-dockerd-0.2.5-3.el8.x86_64.rpm

rpm -ivh cri-dockerd-0.2.5-3.el8.x86_64.rpm

for i in{k8s-master2,k8s-master3,k8s-worker1};do scp cri-dockerd-0.2.5-3.el8.x86_64.rpm $i: ; ssh $i "rpm -ivh cri-dockerd-0.2.5-3.el8.x86_64.rpm";done

配置 cri-dockerd

#众所周知的原因,从国内 cri-dockerd 服务无法下载 k8s.gcr.io上面相关镜像,导致无法启动,所以需要修改cri-dockerd 使用国内镜像源。
sed -ri '/ExecStart.*/s@(ExecStart.*)@\1 --pod-infra-container-image registry.aliyuncs.com/google_containers/pause:3.8@g' /lib/systemd/system/cri-docker.service

systemctl daemon-reload && systemctl enable --now cri-docker

for i in {k8s-master2,k8s-master3,k8s-worker1};do scp /lib/systemd/system/cri-docker.service $i:/lib/systemd/system/cri-docker.service; ssh $i "systemctl daemon-reload && systemctl enable --now cri-docker.service";done

5.2.2 二进制包方式安装cri-dockerd

#下载安装包
wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.2.5/cri-dockerd-0.2.5.amd64.tgz
#解压
tar xf cri-dockerd-0.2.5.amd64.tgz
#直接
cp cri-dockerd/cri-dockerd /usr/bin/
chmod +x /usr/bin/cri-dockerd


cat > /usr/lib/systemd/system/cri-docker.service << 'EOF'
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd://
#ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.8
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

#生成socket文件
cat > /usr/lib/systemd/system/cri-docker.socket << 'EOF'
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target
EOF

sed -ri '/ExecStart.*/s@(ExecStart.*)@\1 --pod-infra-container-image registry.aliyuncs.com/google_containers/pause:3.8@g' /lib/systemd/system/cri-docker.service

systemctl daemon-reload && systemctl enable --now cri-docker
systemctl is-active cri-docke

#拷贝到其他主机
for i in {k8s-master2,k8s-master3,k8s-worker1};do scp /lib/systemd/system/cri-docker.service $i:/lib/systemd/system/cri-docker.service; ssh $i "systemctl daemon-reload && systemctl enable --now cri-docker.service";done

6. 安装Containerd作为Container Runtime

ctr是containerd本身的CLI工具。

crictl是kubernetes社区定义的专门CLI工具。

6.1 所有k8s节点安装Containerd

#下载cni安装包
wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz

#创建cni插件所需目录
mkdir -p /etc/cni/net.d /opt/cni/bin 

#安装cni插件，必须将cni解压到/opt/cni/bin目录，否则nerdctl为容器映射端口时，会出现找不到cni插件的报错
#解压cni二进制包，直接解压到指定目录就可以
tar xf cni-plugins-linux-amd64-v1.1.1.tgz -C /opt/cni/bin/

#下载cri-containerd安装包
wget https://github.com/containerd/containerd/releases/download/v1.6.9/cri-containerd-cni-1.6.9-linux-amd64.tar.gz

#直接解压到根目录下，默认解压后会有如下目录：etc、opt、usr会把对应的目解压到 / 下对应目录中，这样就省去复制文件步骤
tar -xzf cri-containerd-cni-1.6.9-linux-amd64.tar.gz -C /


#创建服务启动文件
cat > /etc/systemd/system/containerd.service << 'EOF'
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=infinity
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF

6.2 生成配置文件并修改

# 创建默认配置文件
mkdir -p /etc/containerd

containerd config default >/etc/containerd/config.toml

# ls /etc/containerd/
config.toml

# 修改Containerd的配置文件，把SystemdCgroup = false修改为：SystemdCgroup = true
sed -i 's@systemd_cgroup = false@systemd_cgroup = true@' /etc/containerd/config.toml

# 修改sandbox_image 镜像源，1.24以下k8s.gcr.io 、1.25 改成了registry.k8s.io
sed -i '[email protected]/pause:[email protected]/google_containers/pause:3.7@' /etc/containerd/config.toml

# 查看是否修改
grep -iE 'systemd_cgroup|sandbox_image' /etc/containerd/config.toml

# Containerd配置镜像加速（可选）
# endpoint位置添加阿里云的镜像源
vi /etc/containerd/config.toml
    [plugins."io.containerd.grpc.v1.cri".registry]
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://xxxxxxxx.mirror.aliyuncs.com" ,"https://registry-1.docker.io"]


# 启动并设置为开机启动
systemctl daemon-reload
systemctl enable --now containerd

6.3 配置crictl客户端连接的运行时位置

#下载crictl安装包
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz

#解压
tar xf crictl-v1.25.0-linux-amd64.tar.gz -C /usr/bin/

#生成配置文件
cat > /etc/crictl.yaml << 'EOF'
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

#测试
systemctl restart containerd
crictl info
crictl ps

6.4 所有节点安装runc

由于上述软件包中包含的runc对系统依赖过多，所以建议单独下载安装。
默认runc执行时提示：runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond

#下载runc安装包
wget https://github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64
chmod +x runc.amd64

#替换掉原软件包中的runc
whereis runc  #查找出runc所在路径
mv runc.amd64 /usr/local/sbin/runc

# runc -v
runc version 1.1.1
commit: v1.1.1-0-g067aaf85
spec: 1.0.2-dev
go: go1.17.6
libseccomp: 2.5.3

#重启containerd
systemctl restart containerd

# nerdctl安装，可以不安装（可选）
tar zxvf nerdctl-1.0.0-linux-amd64.tar.gz
cp nerdctl /usr/local/bin/

7. 部署ETCD集群

在k8s-master1上操作.

7.1 创建工作目录

mkdir -p /data/k8s-work

7.2 获取cfssl工具

cd /data/k8s-work
#下载证书生成工具cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64

说明：
cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有：
- cfssl，是CFSSL的命令行工具。
- cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。

chmod +x cfssl*

mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

cfssl version

7.3 创建CA证书

7.3.1 配置ca证书请求文件

各种 CA 证书类型：

https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/auth.md

cat > ca-csr.json << "EOF"
{
  "CN": "kubernetes",
  "key": {
      "algo": "rsa",
      "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ],
  "ca": {
          "expiry": "87600h"
  }
}
EOF

7.3.2 创建ca证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

7.3.3 配置ca证书策略

cfssl print-defaults config > ca-config.json

cat > ca-config.json << "EOF"
{
  "signing": {
      "default": {
          "expiry": "87600h"
        },
      "profiles": {
          "kubernetes": {
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          }
      }
  }
}
EOF

# server auth :表示client可以使用该ca对server提供的证书进行验证。
# client auth :表示server可以使用该ca对client提供的证书进行验证。

7.4 创建etcd证书

7.4.1 配置etcd请求文件

cat > etcd-csr.json << "EOF"
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.3.111",
    "192.168.3.112",
    "192.168.3.113"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [{
    "C": "CN",
    "ST": "Beijing",
    "L": "Beijing",
    "O": "kubemsb",
    "OU": "CN"
  }]
}
EOF

7.4.2 生成etcd证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

# ls
输出:
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

7.5 部署etcd集群

7.5.1 下载etcd软件包

在github官方网站中搜索 “etcd”，在 “Release” 中选择需要安装的版本。

wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz

7.5.2 安装etcd软件

tar -xvf etcd-v3.5.2-linux-amd64.tar.gz
cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/

7.5.3 分发etcd软件

scp etcd-v3.5.2-linux-amd64/etcd* k8s-master2:/usr/local/bin/

scp etcd-v3.5.2-linux-amd64/etcd* k8s-master3:/usr/local/bin/

7.5.4 创建etcd配置文件

mkdir /etc/etcd

cat >  /etc/etcd/etcd.conf << "EOF"
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.3.111:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.3.111:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.3.111:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.3.111:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.3.111:2380,etcd2=https://192.168.3.112:2380,etcd3=https://192.168.3.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

说明：
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

7.5.5 创建etcd服务配置文件

mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd

cd /data/k8s-work
cp ca*.pem /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl

cat > /etc/systemd/system/etcd.service << "EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
  --cert-file=/etc/etcd/ssl/etcd.pem \
  --key-file=/etc/etcd/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
  --peer-client-cert-auth \
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

7.5.6 同步etcd配置到集群其它master节点

#其他master节点上创建目录
mkdir -p /etc/etcd
mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd

#从k8s-master1上复制服务配置文件,注意需要修改etcd节点名称及IP地址
for i in k8s-master2 k8s-master3 ;
do scp /etc/etcd/etcd.conf $i:/etc/etcd/ ;
done

# k8s-master2:

cat /etc/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.3.112:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.3.112:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.3.112:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.3.112:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.3.111:2380,etcd2=https://192.168.3.112:2380,etcd3=https://192.168.3.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

# k8s-master3:

cat /etc/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.3.113:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.3.113:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.3.113:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.3.113:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.3.111:2380,etcd2=https://192.168.3.112:2380,etcd3=https://192.168.3.113:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

#从k8s-master1上复制证书文件
for i in k8s-master2 k8s-master3 ;
do scp /etc/etcd/ssl/* $i:/etc/etcd/ssl ;
done

#从k8s-master1上复制服务启动配置文件
for i in k8s-master2 k8s-master3 ;
do scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/ ;
done

7.5.7 启动etcd集群

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

7.5.8 验证etcd集群状态

# 检查ETCD数据库集群状态：
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 endpoint health

+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.3.113:2379 |   true | 10.393062ms |       |
| https://192.168.3.111:2379 |   true |  15.70437ms |       |
| https://192.168.3.112:2379 |   true | 15.871684ms |       |
+----------------------------+--------+-------------+-------+

# 检查ETCD数据库性能：
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 check perf

59 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooom  !  98.33%
PASS: Throughput is 151 writes/s
PASS: Slowest request took 0.066478s
PASS: Stddev is 0.002354s
PASS

# 检查ETCD数据库集群
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 member list

+------------------+---------+-------+----------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+----------------------------+----------------------------+------------+
| 9b449b0ff1d4c375 | started | etcd1 | https://192.168.3.111:2380 | https://192.168.3.111:2379 |      false |
| d1fbb74bc6a61e5c | started | etcd2 | https://192.168.3.112:2380 | https://192.168.3.112:2379 |      false |
| f60b205fb02fe23c | started | etcd3 | https://192.168.3.113:2380 | https://192.168.3.113:2379 |      false |
+------------------+---------+-------+----------------------------+----------------------------+------------+

# 检查ETCD数据库集群认证:
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 endpoint status

+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT          |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.3.111:2379 | 9b449b0ff1d4c375 |   3.5.2 |   24 MB |      true |      false |         2 |     403774 |             403774 |        |
| https://192.168.3.112:2379 | d1fbb74bc6a61e5c |   3.5.2 |   24 MB |     false |      false |         2 |     403774 |             403774 |        |
| https://192.168.3.113:2379 | f60b205fb02fe23c |   3.5.2 |   24 MB |     false |      false |         2 |     403774 |             403774 |        |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

8. Kubernetes master集群部署

8.1 Kubernetes软件包下载

打开github.com网站下载： https://github.com/kubernetes/kubernetes/
wget https://dl.k8s.io/v1.25.0/kubernetes-server-linux-amd64.tar.gz

8.2 Kubernetes master节点软件包安装

tar -xvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

8.3 Kubernetes master节点软件分发

将master节点需要的软件包进行分发

scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master2:/usr/local/bin/
scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master3:/usr/local/bin/

8.4 在集群节点上创建目录

所有节点上创建kubernets目录

mkdir -p /etc/kubernetes/ 
mkdir -p /etc/kubernetes/ssl 
mkdir -p /var/log/kubernetes

8.5 部署api-server

8.5.1 创建apiserver证书请求文件

cat > kube-apiserver-csr.json << "EOF"
{
"CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.3.111",
    "192.168.3.112",
    "192.168.3.113",
    "192.168.3.114",
    "192.168.3.115",
    "192.168.3.100",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ]
}
EOF

说明：
1.如果 hosts 字段不为空则需要指定授权使用该证书的 IP（含VIP） 或域名列表。由于该证书被集群使用，需要将节点的IP都填上，为了方便后期扩容可以多写几个预留的IP。
2.同时还需要填写 service 网络的首个IP(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.96.0.1)。

8.5.2 生成apiserver证书及token文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

cat > token.csv << "EOF"
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

说明：创建TLS机制所需TOKEN 
TLS Bootstraping：Master apiserver启用TLS认证后，Node节点kubelet和kube-proxy与kube-apiserver进行通信，必须使用CA签发的有效证书才可以，当Node节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes引入了TLS bootstraping机制来自动颁发客户端证书，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。所以强烈建议在Node上使用这种方式，目前主要用于kubelet，kube-proxy还是由我们统一颁发一个证书。

8.5.3 创建apiserver服务配置文件

cat > /etc/kubernetes/kube-apiserver.conf << "EOF"
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.3.111 \
  --secure-port=6443 \
  --advertise-address=192.168.3.111 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"
EOF

8.5.4 创建建apiserver服务管理配置文件

cat > /etc/systemd/system/kube-apiserver.service << "EOF"
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service

[Service]
EnvironmentFile=/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

8.5.5 同步文件到集群master节点

#将生成的证书文件拷贝到master1节点对应目录下
cp ca*.pem /etc/kubernetes/ssl/
cp kube-apiserver*.pem /etc/kubernetes/ssl/
cp token.csv /etc/kubernetes/

#同步证书文件到其他master节点
scp /etc/kubernetes/token.csv k8s-master2:/etc/kubernetes
scp /etc/kubernetes/token.csv k8s-master3:/etc/kubernetes

scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master2:/etc/kubernetes/ssl
scp /etc/kubernetes/ssl/kube-apiserver*.pem k8s-master3:/etc/kubernetes/ssl

scp /etc/kubernetes/ssl/ca*.pem k8s-master2:/etc/kubernetes/ssl
scp /etc/kubernetes/ssl/ca*.pem k8s-master3:/etc/kubernetes/ssl

#同步apiserver配置文件到其他master节点，注意修改对应IP地址
scp /etc/kubernetes/kube-apiserver.conf k8s-master2:/etc/kubernetes/kube-apiserver.conf

#修改master2节点上的配置文件中对应的IP地址
# cat /etc/kubernetes/kube-apiserver.conf
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.3.112 \
  --secure-port=6443 \
  --advertise-address=192.168.3.112 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"

scp /etc/kubernetes/kube-apiserver.conf k8s-master3:/etc/kubernetes/kube-apiserver.conf

#修改master3节点上的配置文件中对应的IP地址
# cat /etc/kubernetes/kube-apiserver.conf
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --anonymous-auth=false \
  --bind-address=192.168.3.113 \
  --secure-port=6443 \
  --advertise-address=192.168.3.113 \
  --authorization-mode=Node,RBAC \
  --runtime-config=api/all=true \
  --enable-bootstrap-token-auth \
  --service-cluster-ip-range=10.96.0.0/16 \
  --token-auth-file=/etc/kubernetes/token.csv \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \
  --service-account-issuer=api \
  --etcd-cafile=/etc/etcd/ssl/ca.pem \
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
  --etcd-servers=https://192.168.3.111:2379,https://192.168.3.112:2379,https://192.168.3.113:2379 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/kube-apiserver-audit.log \
  --event-ttl=1h \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=4"

#同步apiserver服务管理文件到其他master节点
scp /etc/systemd/system/kube-apiserver.service k8s-master2:/etc/systemd/system/kube-apiserver.service
scp /etc/systemd/system/kube-apiserver.service k8s-master3:/etc/systemd/system/kube-apiserver.service

8.5.6 启动apiserver服务

# 启动
systemctl daemon-reload
systemctl enable --now kube-apiserver

systemctl status kube-apiserver

# 测试
curl --insecure https://192.168.3.111:6443/
curl --insecure https://192.168.3.112:6443/
curl --insecure https://192.168.3.113:6443/
curl --insecure https://192.168.3.100:6443/

8.6 部署kubectl命令行工具

8.6.1 创建kubectl证书请求文件

kubectl 与 apiserver https 安全端口通信，apiserver 对提供的证书进行认证和授权。
kubectl 作为集群的管理工具，需要被授予最高权限。这里创建具有最高权限的 admin 证书。

cat > admin-csr.json << "EOF"
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "system:masters",             
      "OU": "system"
    }
  ]
}
EOF

说明：
后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；
1. kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group 为system:masters 与 Role cluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；
2. O指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；
注意：
这个admin 证书，是将来生成管理员用的kubeconfig配置文件用的，现在我们一般建议使用RBAC来对kubernetes 进行角色权限控制，kubernetes将证书中的 CN 字段作为User， O 字段作为 Group；
•	"O": "system:masters", 必须是system:masters，否则后面kubectl create clusterrolebinding报错。
•	预定义的 ClusterRoleBinding cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定，该 Role 授予所有 API的权限；
•	该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；

8.6.2 生成kubectl证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

8.6.3 复制证书文件到指定目录

cp admin*.pem /etc/kubernetes/ssl/

#复制到其他master节点
scp admin*.pem k8s-master2:/etc/kubernetes/ssl/
scp admin*.pem k8s-master3:/etc/kubernetes/ssl/

8.6.4 生成kubeconfig配置文件

kube.config 为 kubectl 的配置文件，包含访问 apiserver 的所有信息，如 apiserver 地址、CA 证书和自身使用的证书。

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.3.100:6443 --kubeconfig=kube.config

kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config

kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config

kubectl config use-context kubernetes --kubeconfig=kube.config

说明：
•	--certificate-authority：验证 kube-apiserver 证书的根证书；
•	--client-certificate、--client-key：刚生成的 admin 证书和私钥，连接 kube-apiserver 时使用；
•	--embed-certs=true：将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl.kubeconfig 文件中(不加时，写入的是证书文件路径)；

8.6.5 准备kubectl配置文件并进行角色绑定

mkdir ~/.kube
cp kube.config ~/.kube/config
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes --kubeconfig=/root/.kube/config

#拷贝到其他master节点
mkdir ~/.kube
scp kube.config  k8s-master2:~/.kube/config
scp kube.config  k8s-master3:~/.kube/config

8.6.6 看集群状态

export KUBECONFIG=$HOME/.kube/config

#查看集群信息
kubectl cluster-info

#查看集群组件状态
kubectl get componentstatuses

#查看命名空间中资源对象
kubectl get all --all-namespaces

8.6.7 同步kubectl配置文件到集群其它master节点

k8s-master2:
mkdir /root/.kube

k8s-master3:
mkdir /root/.kube

scp /root/.kube/config k8s-master2:/root/.kube/config
scp /root/.kube/config k8s-master3:/root/.kube/config

8.6.8 配置kubectl命令补全(可选)

yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
kubectl completion bash > ~/.kube/completion.bash.inc
source '/root/.kube/completion.bash.inc'  
source $HOME/.bash_profile

8.7 部署kube-controller-manager

8.7.1 创建kube-controller-manager证书请求文件

cat > kube-controller-manager-csr.json << "EOF"
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.3.111",
      "192.168.3.112",
      "192.168.3.113"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
EOF

说明：
hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 为 system:kube-controller-manager;
O 为 system:kube-controller-manager，kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限;

8.7.2 创建kube-controller-manager证书文件

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

# ls
kube-controller-manager.csr     
kube-controller-manager-csr.json
kube-controller-manager-key.pem
kube-controller-manager.pem

8.7.3 创建kube-controller-manager的kube-controller-manager.kubeconfig

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.3.100:6443 --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig

kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig

8.7.4 创建kube-controller-manager配置文件

cat > kube-controller-manager.conf << "EOF"
KUBE_CONTROLLER_MANAGER_OPTS=" \
  --secure-port=10257 \
  --bind-address=127.0.0.1 \
  --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
  --service-cluster-ip-range=10.96.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --allocate-node-cidrs=true \
  --cluster-cidr=10.244.0.0/16 \
  --cluster-signing-duration=87600h \
  --root-ca-file=/etc/kubernetes/ssl/ca.pem \
  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
  --leader-elect=true \
  --feature-gates=RotateKubeletServerCertificate=true \
  --controllers=*,bootstrapsigner,tokencleaner \
  --tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
  --tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
  --use-service-account-credentials=true \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2"
EOF

8.7.5 创建kube-controller-manager服务启动文件

cat > kube-controller-manager.service << "EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

8.7.6 同步文件到集群master节点

#拷贝文件到指定目录
cp kube-controller-manager*.pem /etc/kubernetes/ssl/
cp kube-controller-manager.kubeconfig /etc/kubernetes/
cp kube-controller-manager.conf /etc/kubernetes/
cp kube-controller-manager.service /usr/lib/systemd/system/

#同步到其他master节点
scp  kube-controller-manager*.pem k8s-master2:/etc/kubernetes/ssl/
scp  kube-controller-manager*.pem k8s-master3:/etc/kubernetes/ssl/
scp  kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master2:/etc/kubernetes/
scp  kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master3:/etc/kubernetes/
scp  kube-controller-manager.service k8s-master2:/usr/lib/systemd/system/
scp  kube-controller-manager.service k8s-master3:/usr/lib/systemd/system/

#查看证书
openssl x509 -in /etc/kubernetes/ssl/kube-controller-manager.pem -noout -text

8.7.7 启动kube-controller-manager服务

systemctl daemon-reload 
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

kubectl get componentstatuses

8.8 部署kube-scheduler

8.8.1 创建kube-scheduler证书请求文件

cat > kube-scheduler-csr.json << "EOF"
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.3.111",
      "192.168.3.112",
      "192.168.3.113"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
      {
        "C": "CN",
        "ST": "Beijing",
        "L": "Beijing",
        "O": "system:kube-scheduler",
        "OU": "system"
      }
    ]
}
EOF

8.8.2 生成kube-scheduler证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

# ls
kube-scheduler.csr
kube-scheduler-csr.json
kube-scheduler-key.pem
kube-scheduler.pem

8.8.3 创建kube-scheduler的kubeconfig

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.3.100:6443 --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig

kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig

8.8.4 创建kube-scheduler服务配置文件

cat > kube-scheduler.conf << "EOF"
KUBE_SCHEDULER_OPTS=" \
--bind-address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"
EOF

8.8.5 创建kube-scheduler服务启动配置文件

cat > kube-scheduler.service << "EOF"
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

8.8.6 同步文件至集群master节点

cp kube-scheduler*.pem /etc/kubernetes/ssl/
cp kube-scheduler.kubeconfig /etc/kubernetes/
cp kube-scheduler.conf /etc/kubernetes/
cp kube-scheduler.service /usr/lib/systemd/system/

#同步到其他master节点
scp  kube-scheduler*.pem k8s-master2:/etc/kubernetes/ssl/
scp  kube-scheduler*.pem k8s-master3:/etc/kubernetes/ssl/
scp  kube-scheduler.kubeconfig kube-scheduler.conf k8s-master2:/etc/kubernetes/
scp  kube-scheduler.kubeconfig kube-scheduler.conf k8s-master3:/etc/kubernetes/
scp  kube-scheduler.service k8s-master2:/usr/lib/systemd/system/
scp  kube-scheduler.service k8s-master3:/usr/lib/systemd/system/

8.8.7 启动kube-scheduler服务

systemctl daemon-reload
systemctl enable --now kube-scheduler
systemctl status kube-scheduler

9. Kubernets worker节点部署

9.1 Kubernets worker节点软件分发

将work节点需要的软件包进行分发，安装包全部集成在kubernetes-server-linux-amd64.tar.gz包中，直接去拷贝就行。因为本次是将master也作为work节点，所有也需要安装。

cd kubernetes/server/bin/

scp kubelet kube-proxy k8s-master1:/usr/local/bin
scp kubelet kube-proxy k8s-master2:/usr/local/bin
scp kubelet kube-proxy k8s-master3:/usr/local/bin
scp kubelet kube-proxy k8s-worker1:/usr/local/bin

9.2 部署kubelet

在k8s-master1上操作

9.2.1 创建目录和kubelet-bootstrap.kubeconfig

#所有节点创建kubelet所需目录
mkdir -p /var/lib/kubelet
mkdir -p /var/log/kubernetes

#创建kubelet-bootstrap.kubeconfig
BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
echo $BOOTSTRAP_TOKEN

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.3.100:6443 --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=kubelet-bootstrap

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig

kubectl describe clusterrolebinding cluster-system-anonymous

kubectl describe clusterrolebinding kubelet-bootstrap

9.2.2 创建kubelet配置文件

cat > kubelet.json << "EOF"
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "192.168.3.111",
  "port": 10250,
  "readOnlyPort": 10255,
  "cgroupDriver": "systemd",                    
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "clusterDomain": "cluster.local.",
  "clusterDNS": ["10.96.0.2"]
}
EOF

9.2.3 创建kubelet服务启动管理文件

cat > kubelet.service << "EOF"
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.target  #如果是运行docker，修改成docker.target
Requires=containerd.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
  --cert-dir=/etc/kubernetes/ssl \
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
  --config=/etc/kubernetes/kubelet.json \
  --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

9.2.4 同步kubelet文件到集群节点

cp kubelet-bootstrap.kubeconfig /etc/kubernetes/
cp kubelet.json /etc/kubernetes/
cp kubelet.service /usr/lib/systemd/system/

for i in  k8s-master2 k8s-master3 k8s-worker1;do scp kubelet-bootstrap.kubeconfig kubelet.json $i:/etc/kubernetes/;done

for i in  k8s-master2 k8s-master3 k8s-worker1;do scp ca.pem $i:/etc/kubernetes/ssl/;done

for i in  k8s-master2 k8s-master3 k8s-worker1;do scp kubelet.service $i:/usr/lib/systemd/system/;done

说明：
其他节点同步完之后记得将kubelet.json中address需要修改为当前主机IP地址。

9.2.5 启动kubelet服务

systemctl daemon-reload
systemctl enable --now kubelet

systemctl status kubelet

# 查看集群状态
# kubectl get nodes
NAME          STATUS     ROLES    AGE     VERSION
k8s-master1   NotReady   <none>   2m55s   v1.25.0
k8s-master2   NotReady   <none>   45s     v1.25.0
k8s-master3   NotReady   <none>   39s     v1.25.0
k8s-worker1   NotReady   <none>   5m1s    v1.25.0

# kubectl get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR           CONDITION
csr-b949p   7m55s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued
csr-c9hs4   3m34s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued
csr-r8vhp   5m50s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued
csr-zb4sr   3m40s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Approved,Issued

9.2 部署kube-proxy

9.2.1 创建kube-proxy证书请求文件

cat > kube-proxy-csr.json << "EOF"
{
  "CN": "system:kube-proxy",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "kubemsb",
      "OU": "CN"
    }
  ]
}
EOF

9.2.2 生成kube-proxy证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

# ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

9.2.3 创建kubeconfig文件

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.3.100:6443 --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

9.2.4 创建kube-proxy服务配置文件

cat > kube-proxy.yaml << "EOF"
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.3.111
clientConnection:
  kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 10.244.0.0/16
healthzBindAddress: 192.168.3.111:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.3.111:10249
mode: "ipvs"
EOF

9.2.5 创建kube-proxy服务启动管理文件

cat >  kube-proxy.service << "EOF"
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
  --config=/etc/kubernetes/kube-proxy.yaml \
  --alsologtostderr=true \
  --logtostderr=false \
  --log-dir=/var/log/kubernetes \
  --v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

9.2.6 同步文件到集群工作节点主机

cp kube-proxy*.pem /etc/kubernetes/ssl/
cp kube-proxy.kubeconfig kube-proxy.yaml /etc/kubernetes/
cp kube-proxy.service /usr/lib/systemd/system/

#同步到其他节点
for i in k8s-master2 k8s-master3 k8s-worker1; do scp kube-proxy.kubeconfig kube-proxy.yaml $i:/etc/kubernetes/;done
for i in k8s-master2 k8s-master3 k8s-worker1; do scp kube-proxy.service $i:/usr/lib/systemd/system/;done

说明：
需要在其他节点上修改kube-proxy.yaml中IP地址为当前主机IP。

9.2.7 服务启动

#所有节点创建对应目录
mkdir -p /var/lib/kube-proxy

systemctl daemon-reload
systemctl enable --now kube-proxy

systemctl status kube-proxy

10. 网络组件部署 Calico

10.1 下载Calico的yaml文件

在线下载配置文件地址是： https://docs.projectcalico.org/manifests/calico.yaml
也可下载其他版本：curl -O https://docs.projectcalico.org/archive/v3.19/manifests/calico.yaml

wget https://docs.projectcalico.org/v3.19/manifests/calico.yaml
或者：
wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate

#查看需要的镜像
cat calico.yaml | grep image

10.2 修改Calico的yaml文件

该yaml文件中 CIDR 默认是192.168.0.0/16，需要与初始化时kube-config.yaml中的配置一致，我的是kubeadm部署的k8s，使用的是10.244.0.0/16
            - name: CALICO_IPV4POOL_CIDR
              value: "10.244.0.0/16"
指定网卡，我的网卡是 ens33
            - name: IP_AUTODETECTION_METHOD
              value: "interface=ens33"

10.3 部署Calico

kubectl create -f calico.yaml

10.4 验证结果

# kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7cc8dd57d9-tf2m5   1/1     Running   0          72s
kube-system   calico-node-llw5w                          1/1     Running   0          72s
kube-system   calico-node-mhh6g                          1/1     Running   0          72s
kube-system   calico-node-twj99                          1/1     Running   0          72s
kube-system   calico-node-zh6xl                          1/1     Running   0          72s

# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
k8s-master1   Ready    <none>   55m   v1.21.10
k8s-master2   Ready    <none>   53m   v1.21.10
k8s-master3   Ready    <none>   53m   v1.21.10
k8s-worker1   Ready    <none>   57m   v1.21.10

11. 部署CoreDNS

cat >  coredns.yaml << "EOF"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    - pods
    - namespaces
    verbs:
    - list
    - watch
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
          lameduck 5s
        }
        ready
        kubernetes cluster.local  in-addr.arpa ip6.arpa {
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  # replicas: not specified here:
  # 1. Default is 1.
  # 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: coredns
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             podAffinityTerm:
               labelSelector:
                 matchExpressions:
                   - key: k8s-app
                     operator: In
                     values: ["kube-dns"]
               topologyKey: kubernetes.io/hostname
      containers:
      - name: coredns
        image: coredns/coredns:1.8.4
        imagePullPolicy: IfNotPresent
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        args: [ "-conf", "/etc/coredns/Corefile" ]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          successThreshold: 1
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: coredns
            items:
            - key: Corefile
              path: Corefile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  annotations:
    prometheus.io/port: "9153"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: 10.96.0.2
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP

EOF

#应用文件
kubectl apply -f coredns.yaml

# kubectl get pods -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7cc8dd57d9-tf2m5   1/1     Running   0          4m7s
kube-system   calico-node-llw5w                          1/1     Running   0          4m7s
kube-system   calico-node-mhh6g                          1/1     Running   0          4m7s
kube-system   calico-node-twj99                          1/1     Running   0          4m7s
kube-system   calico-node-zh6xl                          1/1     Running   0          4m7s
kube-system   coredns-675db8b7cc-ncnf6                   1/1     Running   0          26s

12. 部署应用验证

cat >  nginx.yaml  << "EOF"
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-web
spec:
  replicas: 2
  selector:
    name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.19.6
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-nodeport
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30001
      protocol: TCP
  type: NodePort
  selector:
    name: nginx
EOF

# 应用文件
kubectl apply -f nginx.yaml

# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
nginx-web-qzvw4   1/1     Running   0          58s   10.244.194.65   k8s-worker1   <none>           <none>
nginx-web-spw5t   1/1     Running   0          58s   10.244.224.1    k8s-master2   <none>           <none>

# kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-web-qzvw4   1/1     Running   0          2m2s
pod/nginx-web-spw5t   1/1     Running   0          2m2s

NAME                                     DESIRED   CURRENT   READY   AGE
replicationcontroller/nginx-web   2         2         2       2m2s

NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes               ClusterIP   10.96.0.1       <none>        443/TCP        3h37m
service/nginx-service-nodeport   NodePort    10.96.165.114   <none>        80:30001/TCP   2m2s

12. 部署应用验证

cat >  nginx.yaml  << "EOF"
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-web
spec:
  replicas: 2
  selector:
    name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.19.6
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service-nodeport
spec:
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30001
      protocol: TCP
  type: NodePort
  selector:
    name: nginx
EOF

# 应用文件
kubectl apply -f nginx.yaml

#kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
nginx-web-qzvw4   1/1     Running   0          58s   10.244.194.65   k8s-worker1   <none>           <none>
nginx-web-spw5t   1/1     Running   0          58s   10.244.224.1    k8s-master2   <none>           <none>

#kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-web-qzvw4   1/1     Running   0          2m2s
pod/nginx-web-spw5t   1/1     Running   0          2m2s

NAME                                     DESIRED   CURRENT   READY   AGE
replicationcontroller/nginx-web   2         2         2       2m2s

NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes               ClusterIP   10.96.0.1       <none>        443/TCP        3h37m
service/nginx-service-nodeport   NodePort    10.96.165.114   <none>        80:30001/TCP   2m2s