sealos部署k8s集群

一、在线安装k8s集群

1、准备服务器

服务器配置尽量高一点如8C32G，防止后续扩容配置重启服务器导致服务的短暂不可用
生产环境master节点至少三个
master节点相对于node节点，配置可以稍微低一点如4C16G，但master节点服务器配置要一致

2、服务器配置

初始化配置
安装环境准备：下面的操作需要在所有的节点上执行。

# 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

# 关闭selinux
sed -i 's/enforcing/disabled/' /etc/selinux/config  # 永久
setenforce 0  # 临时

# 关闭swap
swapoff -a  # 临时
sed -ri 's/.*swap.*/#&/' /etc/fstab    # 永久

# 根据规划设置主机名
hostnamectl set-hostname <hostname>

# 在master添加hosts
cat >> /etc/hosts << EOF
192.168.150.138 k8s.master01
192.168.150.136 k8s.node01
192.168.150.137 k8s.node02
EOF

# 将桥接的IPv4流量传递到iptables的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system  # 生效

# 时间同步
yum install ntpdate -y
ntpdate time.windows.com

设置master节点到node节点的免密登录

#生成密钥对
[root@k8s-master01 ~] ssh-keygen #一路按回车即可，会在当前用户.ssh目录下生产相应秘钥文件

#拷贝公钥到master、node节点
[root@k8s-master01 ~] ssh-copy-id -i .ssh/id_rsa.pub [email protected]
[root@k8s-master01 ~] ssh-copy-id -i .ssh/id_rsa.pub [email protected]
[root@k8s-master01 ~] ssh-copy-id -i .ssh/id_rsa.pub [email protected]

3、二进制安装sealos

wget https://github.com/labring/sealos/releases/download/v4.1.3/sealos_4.1.3_linux_amd64.tar.gz \
   && tar zxvf sealos_4.1.3_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin

4、自定义安装配置

运行 sealos gen 生成一个 Clusterfile，例如：
基于docker。
为了方便配置harbor私有仓库，我这边使用的基于docker的版本镜像

sealos run labring/kubernetes-docker:v1.22.15 labring/helm:v3.8.2 labring/calico:v3.24.1 \
--masters 192.168.150.138 \
--nodes 192.168.150.136,192.168.150.137\
--passwd 123456  > Clusterfile

默认使用containerd

$ sealos gen labring/kubernetes:v1.22.15 labring/helm:v3.8.2 labring/calico:v3.24.1 \
   --masters 192.168.150.138 \
   --nodes 192.168.150.136,192.168.150.137 \
   --passwd 12345678 > Clusterfile

改用docker

sealos run labring/kubernetes-docker:v1.22.15 labring/helm:v3.8.2 labring/calico:v3.24.1 \
	--masters 10.140.19.201 \
	--nodes 10.140.19.202,10.140.19.205,10.140.19.206 \
	--passwd 12345678

生成的 Clusterfile 如下：

apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  creationTimestamp: null
  name: default
spec:
  hosts:
  - ips:
    - 192.168.150.138:22
    roles:
    - master
    - amd64
  - ips:
    - 192.168.150.136:22
    - 192.168.150.137:22
    roles:
    - node
    - amd64
  image:
  - labring/kubernetes-docker:v1.22.15
  - labring/calico:v3.24.1
  ssh:
    passwd: 123456
    pk: /root/.ssh/id_rsa
    port: 22
    user: root
status: {}

将 calico Clusterfile 追加到生成的 Clusterfile 后，然后更新集群配置。例如，要修改 pods 的 CIDR 范围，就可以修改 networking.podSubnet 和 spec.data.spec.calicoNetwork.ipPools.cidr 字段。最终的 Clusterfile 会像是这样:

apiVersion: apps.sealos.io/v1beta1
kind: Cluster
metadata:
  creationTimestamp: null
  name: default
spec:
  hosts:
  - ips:
    - 192.168.150.138:22
    roles:
    - master
    - amd64
  - ips:
    - 192.168.150.136:22
    - 192.168.150.137:22
    roles:
    - node
    - amd64
  image:
  - labring/kubernetes:v1.22.15
  - labring/calico:v3.24.1
  ssh:
    passwd: 123456
    pk: /root/.ssh/id_rsa
    port: 22
    user: root
status: {}
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
networking:
  podSubnet: 10.160.0.0/12
---
apiVersion: apps.sealos.io/v1beta1
kind: Config
metadata:
  name: calico
spec:
  path: manifests/calico.yaml
  data: |
    apiVersion: operator.tigera.io/v1
    kind: Installation
    metadata:
      name: default
    spec:
      # Configures Calico networking.
      calicoNetwork:
        # Note: The ipPools section cannot be modified post-install.
        ipPools:
        - blockSize: 26
          # Note: Must be the same as podCIDR
          cidr: 10.160.0.0/12
          encapsulation: IPIP
          natOutgoing: Enabled
          nodeSelector: all()
        nodeAddressAutodetectionV4:
          interface: "eth.*|en.*"

运行 sealos apply -f Clusterfile 启动集群。集群运行成功后会把 Clusterfile 保存到 .sealos/default/Clusterfile 文件中，可以修改其中字段来重新 apply 对集群进行变更。

注意：

可以参考官方文档或运行 kubeadm config print init-defaults 命令来打印 kubeadm 配置。
实验性功能使用方法请查看 CLI。

5、开始安装集群

[root@k8s-master01 ~]# sealos apply -f Clusterfile

整个过程持续10分钟左右：
输出内容如下：

2022-12-03T13:21:11 info Start to create a new cluster: master [192.168.150.138], worker [192.168.150.136 192.168.150.137]
2022-12-03T13:21:11 info Executing pipeline Check in CreateProcessor.
2022-12-03T13:21:11 info checker:hostname [192.168.150.138:22 192.168.150.136:22 192.168.150.137:22]
2022-12-03T13:21:12 info checker:timeSync [192.168.150.138:22 192.168.150.136:22 192.168.150.137:22]
2022-12-03T13:21:12 info Executing pipeline PreProcess in CreateProcessor.
Resolving "labring/kubernetes" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/labring/kubernetes:v1.24...
Getting image source signatures
Copying blob 492bb2eddf21 done
Copying config 160c217560 done
Writing manifest to image destination
Storing signatures
160c217560b2bbd5601116f97b80af3a0b816333371ceeb24c8c99b70bd704cd
Resolving "labring/helm" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/labring/helm:v3.8.2...
Getting image source signatures
Copying blob 53a6eade9e7e done
Copying config 1123e8b4b4 done
Writing manifest to image destination
Storing signatures
1123e8b4b455ed291f3ec7273af62e49458fe3dd141f5e7cb2a4243d6284deec
Resolving "labring/calico" using unqualified-search registries (/etc/containers/registries.conf)
Trying to pull docker.io/labring/calico:v3.24.1...
Getting image source signatures
Copying blob f9de59270f64 done
Copying config e2122fc58f done
Writing manifest to image destination
Storing signatures
e2122fc58fd32f1c93ac75da5c473aed746f1ad9b31a73d1f81a0579b96e775b
default-ov02hfcf
default-gavmcjsy
default-krjxdzgj
2022-12-03T13:23:22 info Executing pipeline RunConfig in CreateProcessor.
2022-12-03T13:23:22 info Executing pipeline MountRootfs in CreateProcessor.
[1/1]copying files to 192.168.150.137:22  25% [==>            ] (1/4, 1 it/s) [0s:2s]which: no docker in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
INFO [2022-12-03 13:23:46] >> check root,port,cri success
[1/1]copying files to 192.168.150.137:22  50% [======>        ] (2/4, 39 it/min) [12s:3s]Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /etc/systemd/system/containerd.service.
[1/1]copying files to 192.168.150.136:22  50% [======>        ] (2/4, 38 it/min) [13s:3s] INFO [2022-12-03 13:23:59] >> Health check containerd!
INFO [2022-12-03 13:23:59] >> containerd is running
INFO [2022-12-03 13:23:59] >> init containerd success
Created symlink from /etc/systemd/system/multi-user.target.wants/image-cri-shim.service to /etc/systemd/system/image-cri-shim.service.
INFO [2022-12-03 13:24:00] >> Health check image-cri-shim!
INFO [2022-12-03 13:24:01] >> image-cri-shim is running
INFO [2022-12-03 13:24:01] >> init shim success

* Applying /usr/lib/sysctl.d/00-system.conf ...
  net.bridge.bridge-nf-call-ip6tables = 0
  net.bridge.bridge-nf-call-iptables = 0
  net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
  kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
  kernel.sysrq = 16
  kernel.core_uses_pid = 1
  kernel.kptr_restrict = 1
  net.ipv4.conf.default.rp_filter = 1
  net.ipv4.conf.all.rp_filter = 1
  net.ipv4.conf.default.accept_source_route = 0
  net.ipv4.conf.all.accept_source_route = 0
  net.ipv4.conf.default.promote_secondaries = 1
  net.ipv4.conf.all.promote_secondaries = 1
  fs.protected_hardlinks = 1
  fs.protected_symlinks = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
  net.bridge.bridge-nf-call-ip6tables = 1
  net.bridge.bridge-nf-call-iptables = 1
  net.ipv4.conf.all.rp_filter = 0
* Applying /etc/sysctl.conf ...
  net.ipv4.ip_forward = 1
  Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
  Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
  Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
   INFO [2022-12-03 13:24:06] >> init kube success
   INFO [2022-12-03 13:24:06] >> init containerd rootfs success
  192.168.150.137:22: which: no docker in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
  192.168.150.137:22:  INFO [2022-12-03 13:25:17] >> check root,port,cri success
  192.168.150.136:22: which: no docker in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
  192.168.150.136:22:  INFO [2022-12-03 13:25:31] >> check root,port,cri success
  192.168.150.137:22: Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /etc/systemd/system/containerd.service.
  192.168.150.137:22:  INFO [2022-12-03 13:25:37] >> Health check containerd!
  192.168.150.137:22:  INFO [2022-12-03 13:25:37] >> containerd is running
  192.168.150.137:22:  INFO [2022-12-03 13:25:37] >> init containerd success
  192.168.150.137:22: Created symlink from /etc/systemd/system/multi-user.target.wants/image-cri-shim.service to /etc/systemd/system/image-cri-shim.service.
  192.168.150.137:22:  INFO [2022-12-03 13:25:38] >> Health check image-cri-shim!
  192.168.150.137:22:  INFO [2022-12-03 13:25:38] >> image-cri-shim is running
  192.168.150.137:22:  INFO [2022-12-03 13:25:38] >> init shim success
  192.168.150.137:22: * Applying /usr/lib/sysctl.d/00-system.conf ...
  192.168.150.137:22: net.bridge.bridge-nf-call-ip6tables = 0
  192.168.150.137:22: net.bridge.bridge-nf-call-iptables = 0
  192.168.150.137:22: net.bridge.bridge-nf-call-arptables = 0
  192.168.150.137:22: * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
  192.168.150.137:22: kernel.yama.ptrace_scope = 0
  192.168.150.137:22: * Applying /usr/lib/sysctl.d/50-default.conf ...
  192.168.150.137:22: kernel.sysrq = 16
  192.168.150.137:22: kernel.core_uses_pid = 1
  192.168.150.137:22: kernel.kptr_restrict = 1
  192.168.150.137:22: net.ipv4.conf.default.rp_filter = 1
  192.168.150.137:22: net.ipv4.conf.all.rp_filter = 1
  192.168.150.137:22: net.ipv4.conf.default.accept_source_route = 0
  192.168.150.137:22: net.ipv4.conf.all.accept_source_route = 0
  192.168.150.137:22: net.ipv4.conf.default.promote_secondaries = 1
  192.168.150.137:22: net.ipv4.conf.all.promote_secondaries = 1
  192.168.150.137:22: fs.protected_hardlinks = 1
  192.168.150.137:22: fs.protected_symlinks = 1
  192.168.150.137:22: * Applying /etc/sysctl.d/99-sysctl.conf ...
  192.168.150.137:22: * Applying /etc/sysctl.d/k8s.conf ...
  192.168.150.137:22: net.bridge.bridge-nf-call-ip6tables = 1
  192.168.150.137:22: net.bridge.bridge-nf-call-iptables = 1
  192.168.150.137:22: net.ipv4.conf.all.rp_filter = 0
  192.168.150.137:22: * Applying /etc/sysctl.conf ...
  192.168.150.137:22: net.ipv4.ip_forward = 1
  192.168.150.136:22: Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /etc/systemd/system/containerd.service.
  192.168.150.137:22: Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
  192.168.150.137:22: Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
  192.168.150.136:22:  INFO [2022-12-03 13:25:44] >> Health check containerd!
  192.168.150.136:22:  INFO [2022-12-03 13:25:44] >> containerd is running
  192.168.150.136:22:  INFO [2022-12-03 13:25:44] >> init containerd success
  192.168.150.136:22: Created symlink from /etc/systemd/system/multi-user.target.wants/image-cri-shim.service to /etc/systemd/system/image-cri-shim.service.
  192.168.150.136:22:  INFO [2022-12-03 13:25:47] >> Health check image-cri-shim!
  192.168.150.136:22:  INFO [2022-12-03 13:25:48] >> image-cri-shim is running
  192.168.150.136:22:  INFO [2022-12-03 13:25:48] >> init shim success
  192.168.150.136:22: * Applying /usr/lib/sysctl.d/00-system.conf ...
  192.168.150.136:22: net.bridge.bridge-nf-call-ip6tables = 0
  192.168.150.136:22: net.bridge.bridge-nf-call-iptables = 0
  192.168.150.136:22: net.bridge.bridge-nf-call-arptables = 0
  192.168.150.136:22: * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
  192.168.150.136:22: kernel.yama.ptrace_scope = 0
  192.168.150.136:22: * Applying /usr/lib/sysctl.d/50-default.conf ...
  192.168.150.136:22: kernel.sysrq = 16
  192.168.150.136:22: kernel.core_uses_pid = 1
  192.168.150.136:22: kernel.kptr_restrict = 1
  192.168.150.136:22: net.ipv4.conf.default.rp_filter = 1
  192.168.150.136:22: net.ipv4.conf.all.rp_filter = 1
  192.168.150.136:22: net.ipv4.conf.default.accept_source_route = 0
  192.168.150.136:22: net.ipv4.conf.all.accept_source_route = 0
  192.168.150.136:22: net.ipv4.conf.default.promote_secondaries = 1
  192.168.150.136:22: net.ipv4.conf.all.promote_secondaries = 1
  192.168.150.136:22: fs.protected_hardlinks = 1
  192.168.150.136:22: fs.protected_symlinks = 1
  192.168.150.136:22: * Applying /etc/sysctl.d/99-sysctl.conf ...
  192.168.150.136:22: * Applying /etc/sysctl.d/k8s.conf ...
  192.168.150.136:22: net.bridge.bridge-nf-call-ip6tables = 1
  192.168.150.136:22: net.bridge.bridge-nf-call-iptables = 1
  192.168.150.136:22: net.ipv4.conf.all.rp_filter = 0
  192.168.150.136:22: * Applying /etc/sysctl.conf ...
  192.168.150.136:22: net.ipv4.ip_forward = 1
  192.168.150.136:22: Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
  192.168.150.136:22: Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
  192.168.150.137:22: Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
  192.168.150.137:22:  INFO [2022-12-03 13:26:02] >> init kube success
  192.168.150.137:22:  INFO [2022-12-03 13:26:03] >> init containerd rootfs success
  192.168.150.136:22: Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
  192.168.150.136:22:  INFO [2022-12-03 13:26:20] >> init kube success
  192.168.150.136:22:  INFO [2022-12-03 13:26:20] >> init containerd rootfs success
  2022-12-03T13:26:29 info Executing pipeline Init in CreateProcessor.
  2022-12-03T13:26:29 info start to copy kubeadm config to master0
  2022-12-03T13:26:36 info start to generate cert and kubeConfig...
  2022-12-03T13:26:36 info start to generator cert and copy to masters...
  2022-12-03T13:26:37 info apiserver altNames : {map[apiserver.cluster.local:apiserver.cluster.local k8s-master01:k8s-master01 kubernetes:kubernetes kubernetes.default:kubernetes.default kubernetes.default.svc:kubernetes.default.svc kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local localhost:localhost] map[10.103.97.2:10.103.97.2 10.96.0.1:10.96.0.1 127.0.0.1:127.0.0.1 192.168.150.138:192.168.150.138]}
  2022-12-03T13:26:37 info Etcd altnames : {map[k8s-master01:k8s-master01 localhost:localhost] map[127.0.0.1:127.0.0.1 192.168.150.138:192.168.150.138 ::1:::1]}, commonName : k8s-master01
  2022-12-03T13:26:40 info start to copy etc pki files to masters
  2022-12-03T13:26:40 info start to create kubeconfig...
  2022-12-03T13:26:42 info start to copy kubeconfig files to masters
  2022-12-03T13:26:42 info start to copy static files to masters
  2022-12-03T13:26:42 info start to apply registry
  Created symlink from /etc/systemd/system/multi-user.target.wants/registry.service to /etc/systemd/system/registry.service.
   INFO [2022-12-03 13:26:43] >> Health check registry!
   INFO [2022-12-03 13:26:43] >> registry is running
   INFO [2022-12-03 13:26:43] >> init registry success
  2022-12-03T13:26:43 info start to init master0...
  2022-12-03T13:26:43 info registry auth in node 192.168.150.138:22
  2022-12-03T13:26:44 info domain sealos.hub:192.168.150.138 append success
  2022-12-03T13:26:45 info domain apiserver.cluster.local:192.168.150.138 append success
  W1203 13:26:47.278211    1853 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
  [init] Using Kubernetes version: v1.24.4
  [preflight] Running pre-flight checks
        [WARNING FileExisting-socat]: socat not found in system path
        [WARNING Hostname]: hostname "k8s-master01" could not be reached
        [WARNING Hostname]: hostname "k8s-master01": lookup k8s-master01 on 114.114.114.114:53: no such host
  [preflight] Pulling images required for setting up a Kubernetes cluster
  [preflight] This might take a minute or two, depending on the speed of your internet connection
  [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
  [certs] Using certificateDir folder "/etc/kubernetes/pki"
  [certs] Using existing ca certificate authority
  [certs] Using existing apiserver certificate and key on disk
  [certs] Using existing apiserver-kubelet-client certificate and key on disk
  [certs] Using existing front-proxy-ca certificate authority
  [certs] Using existing front-proxy-client certificate and key on disk
  [certs] Using existing etcd/ca certificate authority
  [certs] Using existing etcd/server certificate and key on disk
  [certs] Using existing etcd/peer certificate and key on disk
  [certs] Using existing etcd/healthcheck-client certificate and key on disk
  [certs] Using existing apiserver-etcd-client certificate and key on disk
  [certs] Using the existing "sa" key
  [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
  [kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/admin.conf"
  [kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/kubelet.conf"
  W1203 13:27:41.625270    1853 kubeconfig.go:249] a kubeconfig file "/etc/kubernetes/controller-manager.conf" exists already but has an unexpected API Server URL: expected: https://192.168.150.138:6443, got: https://apiserver.cluster.local:6443
  [kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/controller-manager.conf"
  W1203 13:27:41.976983    1853 kubeconfig.go:249] a kubeconfig file "/etc/kubernetes/scheduler.conf" exists already but has an unexpected API Server URL: expected: https://192.168.150.138:6443, got: https://apiserver.cluster.local:6443
  [kubeconfig] Using existing kubeconfig file: "/etc/kubernetes/scheduler.conf"
  [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  [kubelet-start] Starting the kubelet
  [control-plane] Using manifest folder "/etc/kubernetes/manifests"
  [control-plane] Creating static Pod manifest for "kube-apiserver"
  [control-plane] Creating static Pod manifest for "kube-controller-manager"
  [control-plane] Creating static Pod manifest for "kube-scheduler"
  [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
  [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
  [kubelet-check] Initial timeout of 40s passed.
  [apiclient] All control plane components are healthy after 68.520377 seconds
  [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
  [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
  [upload-certs] Skipping phase. Please see --upload-certs
  [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
  [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
  [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
  [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
  [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
  [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
  [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
  [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
  [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
  [addons] Applied essential addon: CoreDNS
  [addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join apiserver.cluster.local:6443 --token <value withheld> \
        --discovery-token-ca-cert-hash sha256:547bbf832d660cfbf35191db348a78d5b22b6762fb89cb8cc5dd2a98c158aac5 \
        --control-plane --certificate-key <value withheld>

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join apiserver.cluster.local:6443 --token <value withheld> \
        --discovery-token-ca-cert-hash sha256:547bbf832d660cfbf35191db348a78d5b22b6762fb89cb8cc5dd2a98c158aac5
2022-12-03T13:29:00 info Executing pipeline Join in CreateProcessor.
2022-12-03T13:29:00 info [192.168.150.136:22 192.168.150.137:22] will be added as worker
2022-12-03T13:29:00 info start to get kubernetes token...
2022-12-03T13:29:04 info start to join 192.168.150.137:22 as worker
2022-12-03T13:29:04 info start to copy kubeadm join config to node: 192.168.150.137:22
2022-12-03T13:29:04 info start to join 192.168.150.136:22 as worker
2022-12-03T13:29:04 info start to copy kubeadm join config to node: 192.168.150.136:22
192.168.150.137:22: 2022-12-03T13:29:10 info domain apiserver.cluster.local:10.103.97.2 append success
192.168.150.137:22: 2022-12-03T13:29:12 info domain lvscare.node.ip:192.168.150.137 append success
2022-12-03T13:29:12 info registry auth in node 192.168.150.137:22
192.168.150.137:22: 2022-12-03T13:29:14 info domain sealos.hub:192.168.150.138 append success
2022-12-03T13:29:14 info run ipvs once module: 192.168.150.137:22
192.168.150.136:22: 2022-12-03T13:29:15 info domain apiserver.cluster.local:10.103.97.2 append success
192.168.150.137:22: 2022-12-03T13:29:15 info Trying to add route
192.168.150.137:22: 2022-12-03T13:29:15 info success to set route.(host:10.103.97.2, gateway:192.168.150.137)
2022-12-03T13:29:15 info start join node: 192.168.150.137:22
192.168.150.137:22: W1203 13:29:16.148407    3998 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
192.168.150.137:22: [preflight] Running pre-flight checks
192.168.150.137:22:     [WARNING FileExisting-socat]: socat not found in system path
192.168.150.136:22: 2022-12-03T13:29:16 info domain lvscare.node.ip:192.168.150.136 append success
2022-12-03T13:29:16 info registry auth in node 192.168.150.136:22
192.168.150.136:22: 2022-12-03T13:29:17 info domain sealos.hub:192.168.150.138 append success
2022-12-03T13:29:17 info run ipvs once module: 192.168.150.136:22
192.168.150.136:22: 2022-12-03T13:29:18 info Trying to add route
192.168.150.136:22: 2022-12-03T13:29:18 info success to set route.(host:10.103.97.2, gateway:192.168.150.136)
2022-12-03T13:29:18 info start join node: 192.168.150.136:22
192.168.150.136:22: W1203 13:29:18.916599    3968 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
192.168.150.136:22: [preflight] Running pre-flight checks
192.168.150.136:22:     [WARNING FileExisting-socat]: socat not found in system path
192.168.150.137:22:     [WARNING Hostname]: hostname "k8s-node01" could not be reached
192.168.150.137:22:     [WARNING Hostname]: hostname "k8s-node01": lookup k8s-node01 on 114.114.114.114:53: no such host
192.168.150.137:22: [preflight] Reading configuration from the cluster...
192.168.150.137:22: [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
192.168.150.137:22: [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
192.168.150.137:22: [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
192.168.150.137:22: [kubelet-start] Starting the kubelet
192.168.150.137:22: [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
192.168.150.136:22:     [WARNING Hostname]: hostname "k8s-node02" could not be reached
192.168.150.136:22:     [WARNING Hostname]: hostname "k8s-node02": lookup k8s-node02 on 114.114.114.114:53: no such host
192.168.150.136:22: [preflight] Reading configuration from the cluster...
192.168.150.136:22: [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
192.168.150.136:22: [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
192.168.150.136:22: [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
192.168.150.136:22: [kubelet-start] Starting the kubelet
192.168.150.136:22: [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
192.168.150.137:22:
192.168.150.137:22: This node has joined the cluster:
192.168.150.137:22: * Certificate signing request was sent to apiserver and a response was received.
192.168.150.137:22: * The Kubelet was informed of the new secure connection details.
192.168.150.137:22:
192.168.150.137:22: Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
192.168.150.137:22:
2022-12-03T13:29:39 info succeeded in joining 192.168.150.137:22 as worker
192.168.150.136:22:
192.168.150.136:22: This node has joined the cluster:
192.168.150.136:22: * Certificate signing request was sent to apiserver and a response was received.
192.168.150.136:22: * The Kubelet was informed of the new secure connection details.
192.168.150.136:22:
192.168.150.136:22: Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
192.168.150.136:22:
2022-12-03T13:29:41 info succeeded in joining 192.168.150.136:22 as worker
2022-12-03T13:29:42 info start to sync lvscare static pod to node: 192.168.150.137:22 master: [192.168.150.138:6443]
2022-12-03T13:29:42 info start to sync lvscare static pod to node: 192.168.150.136:22 master: [192.168.150.138:6443]
192.168.150.137:22: 2022-12-03T13:29:43 info generator lvscare static pod is success
192.168.150.136:22: 2022-12-03T13:29:44 info generator lvscare static pod is success
2022-12-03T13:29:44 info Executing pipeline RunGuest in CreateProcessor.
2022-12-03T13:29:44 info guest cmd is cp opt/helm /usr/bin/
2022-12-03T13:29:44 info guest cmd is kubectl create namespace tigera-operator
namespace/tigera-operator created
2022-12-03T13:29:45 info guest cmd is helm install calico charts/calico --namespace tigera-operator
NAME: calico
LAST DEPLOYED: Sat Dec  3 13:29:56 2022
NAMESPACE: tigera-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None
2022-12-03T13:30:03 info succeeded in creating a new cluster, enjoy it!
2022-12-03T13:30:03 info
      ___           ___           ___           ___       ___           ___
     /\  \         /\  \         /\  \         /\__\     /\  \         /\  \
    /::\  \       /::\  \       /::\  \       /:/  /    /::\  \       /::\  \
   /:/\ \  \     /:/\:\  \     /:/\:\  \     /:/  /    /:/\:\  \     /:/\ \  \
  _\:\~\ \  \   /::\~\:\  \   /::\~\:\  \   /:/  /    /:/  \:\  \   _\:\~\ \  \
 /\ \:\ \ \__\ /:/\:\ \:\__\ /:/\:\ \:\__\ /:/__/    /:/__/ \:\__\ /\ \:\ \ \__\
 \:\ \:\ \/__/ \:\~\:\ \/__/ \/__\:\/:/  / \:\  \    \:\  \ /:/  / \:\ \:\ \/__/
  \:\ \:\__\    \:\ \:\__\        \::/  /   \:\  \    \:\  /:/  /   \:\ \:\__\
   \:\/:/  /     \:\ \/__/        /:/  /     \:\  \    \:\/:/  /     \:\/:/  /
    \::/  /       \:\__\         /:/  /       \:\__\    \::/  /       \::/  /
     \/__/         \/__/         \/__/         \/__/     \/__/         \/__/

                  Website :https://www.sealos.io/
                  Address :github.com/labring/sealos

6、查看集群

集群创建完需要等个几分钟才会显示正常

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES           AGE     VERSION
k8s-master01   NotReady   control-plane   2m21s   v1.24.4
k8s-node01     NotReady   <none>          97s     v1.24.4
k8s-node02     NotReady   <none>          94s     v1.24.4
[root@k8s-master01 ~]# kubectl get pods
No resources found in default namespace.
[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES           AGE     VERSION
k8s-master01   Ready      control-plane   3m50s   v1.24.4
k8s-node01     NotReady   <none>          3m6s    v1.24.4
k8s-node02     NotReady   <none>          3m3s    v1.24.4
[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
k8s-master01   Ready    control-plane   9m5s    v1.24.4
k8s-node01     Ready    <none>          8m21s   v1.24.4
k8s-node02     Ready    <none>          8m18s   v1.24.4
[root@k8s-master01 ~]# kubectl get pods
No resources found in default namespace.
[root@k8s-master01 ~]# kubectl get pods -A
NAMESPACE          NAME                                       READY   STATUS    RESTARTS       AGE
calico-apiserver   calico-apiserver-6df8578b5f-4sd8b          1/1     Running   0              4m7s
calico-apiserver   calico-apiserver-6df8578b5f-g62wv          1/1     Running   0              4m7s
calico-system      calico-kube-controllers-5b8957ccd7-jsmzm   1/1     Running   0              9m12s
calico-system      calico-node-4wvsg                          1/1     Running   0              9m13s
calico-system      calico-node-q7t5v                          1/1     Running   0              9m13s
calico-system      calico-node-s668p                          1/1     Running   0              9m13s
calico-system      calico-typha-5cd6db69-jt9ss                1/1     Running   0              9m14s
calico-system      calico-typha-5cd6db69-pdlb5                1/1     Running   0              9m6s
calico-system      csi-node-driver-dqrv9                      2/2     Running   0              7m35s
calico-system      csi-node-driver-m86lh                      2/2     Running   0              6m21s
calico-system      csi-node-driver-w9p77                      2/2     Running   0              5m46s
kube-system        coredns-6d4b75cb6d-vvx5g                   1/1     Running   0              10m
kube-system        coredns-6d4b75cb6d-zzfnp                   1/1     Running   0              10m
kube-system        etcd-k8s-master01                          1/1     Running   0              10m
kube-system        kube-apiserver-k8s-master01                1/1     Running   0              10m
kube-system        kube-controller-manager-k8s-master01       1/1     Running   0              10m
kube-system        kube-proxy-9jk96                           1/1     Running   0              9m59s
kube-system        kube-proxy-sv4nj                           1/1     Running   0              9m56s
kube-system        kube-proxy-x2cfj                           1/1     Running   0              10m
kube-system        kube-scheduler-k8s-master01                1/1     Running   1 (2m6s ago)   10m
kube-system        kube-sealos-lvscare-k8s-node01             1/1     Running   0              9m34s
kube-system        kube-sealos-lvscare-k8s-node02             1/1     Running   0              9m51s
tigera-operator    tigera-operator-6bb888d6fc-5ftwh           1/1     Running   0              9m35s
[root@k8s-master01 ~]# kubectl get pods -A -owide
NAMESPACE          NAME                                       READY   STATUS    RESTARTS        AGE     IP                NODE           NOMINATED NODE   READINESS GATES
calico-apiserver   calico-apiserver-6df8578b5f-4sd8b          1/1     Running   0               4m17s   100.125.152.2     k8s-node02     <none>           <none>
calico-apiserver   calico-apiserver-6df8578b5f-g62wv          1/1     Running   0               4m17s   100.97.125.2      k8s-node01     <none>           <none>
calico-system      calico-kube-controllers-5b8957ccd7-jsmzm   1/1     Running   0               9m22s   100.124.32.129    k8s-master01   <none>           <none>
calico-system      calico-node-4wvsg                          1/1     Running   0               9m23s   192.168.150.137   k8s-node01     <none>           <none>
calico-system      calico-node-q7t5v                          1/1     Running   0               9m23s   192.168.150.138   k8s-master01   <none>           <none>
calico-system      calico-node-s668p                          1/1     Running   0               9m23s   192.168.150.136   k8s-node02     <none>           <none>
calico-system      calico-typha-5cd6db69-jt9ss                1/1     Running   0               9m24s   192.168.150.136   k8s-node02     <none>           <none>
calico-system      calico-typha-5cd6db69-pdlb5                1/1     Running   0               9m16s   192.168.150.137   k8s-node01     <none>           <none>
calico-system      csi-node-driver-dqrv9                      2/2     Running   0               7m45s   100.124.32.130    k8s-master01   <none>           <none>
calico-system      csi-node-driver-m86lh                      2/2     Running   0               6m31s   100.125.152.1     k8s-node02     <none>           <none>
calico-system      csi-node-driver-w9p77                      2/2     Running   0               5m56s   100.97.125.1      k8s-node01     <none>           <none>
kube-system        coredns-6d4b75cb6d-vvx5g                   1/1     Running   0               10m     100.124.32.132    k8s-master01   <none>           <none>
kube-system        coredns-6d4b75cb6d-zzfnp                   1/1     Running   0               10m     100.124.32.131    k8s-master01   <none>           <none>
kube-system        etcd-k8s-master01                          1/1     Running   0               10m     192.168.150.138   k8s-master01   <none>           <none>
kube-system        kube-apiserver-k8s-master01                1/1     Running   0               10m     192.168.150.138   k8s-master01   <none>           <none>
kube-system        kube-controller-manager-k8s-master01       1/1     Running   0               10m     192.168.150.138   k8s-master01   <none>           <none>
kube-system        kube-proxy-9jk96                           1/1     Running   0               10m     192.168.150.137   k8s-node01     <none>           <none>
kube-system        kube-proxy-sv4nj                           1/1     Running   0               10m     192.168.150.136   k8s-node02     <none>           <none>
kube-system        kube-proxy-x2cfj                           1/1     Running   0               10m     192.168.150.138   k8s-master01   <none>           <none>
kube-system        kube-scheduler-k8s-master01                1/1     Running   1 (2m16s ago)   10m     192.168.150.138   k8s-master01   <none>           <none>
kube-system        kube-sealos-lvscare-k8s-node01             1/1     Running   0               9m44s   192.168.150.137   k8s-node01     <none>           <none>
kube-system        kube-sealos-lvscare-k8s-node02             1/1     Running   0               10m     192.168.150.136   k8s-node02     <none>           <none>
tigera-operator    tigera-operator-6bb888d6fc-5ftwh           1/1     Running   0               9m45s   192.168.150.137   k8s-node01     <none>           <none>
[root@k8s-master01 ~]# kubectl cluster-info
Kubernetes control plane is running at https://apiserver.cluster.local:6443
CoreDNS is running at https://apiserver.cluster.local:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

报错解决

问题1：function “semverCompare” not defined

[EROR] Applied to cluster error: render env to rootfs failed: failed to create template: /var/lib/containers/storage/overlay/bbfc1569f981a2d0389cf78e288b7392298a39524b7e5a9ebd94611d23a2dcee/merged/etc/image-cri-shim.yaml.tmpl template: image-cri-shim.yaml.tmpl:21: function "semverCompare" not defined

问题2：exec auth.sh failed

error Applied to cluster error: failed to init exec auth.sh failed exit status 127

解决方式

升级sealos到最新版本

二、离线安装k8s集群

下载离线包

https://pan.baidu.com/s/1fu_l8yL_K6BLpSIugKhvAg?pwd=47f5#list/path=%2Fsharelink33820949-51949982255598%2Fsealos%E7%A6%BB%E7%BA%BF%E5%8C%85%2Farm64&parentPath=%2Fsharelink33820949-51949982255598

下载sealos工具

wget https://github.com/labring/sealos/releases/download/v4.0.0/sealos_4.0.0_linux_amd64.tar.gz

tar xf sealos_4.0.0_linux_amd64.tar.gz
mv sealos  /usr/local/bin/
chmod +x /usr/local/bin/sealos

安装k8s集群

老版

wget 
sealos init --passwd '12345678' \
    --master 10.11.1.2-10.11.1.4 \ 
    --node 10.111.1.5 \
    --pkg-url /root/kube1.23.6.tar.gz \
    --version v1.23.6

参数说明：

passwd 服务器密码 123456
master k8s master节点IP地址 10.11.1.2-10.11.1.4
node k8s node节点IP地址 10.11.1.3
pkg-url 离线资源包地址，支持下载到本地，或者一个远程地址 /root/kube1.23.6.tar.gz
version 资源包对应的版本 v1.23.6
apiserver apiserver的虚拟IP地址与域名映射。配置在/etc/hosts中默认值"apiserver.cluster.local"
cert-sans kubernetes apiServerCertSANs ex. 47.0.0.22 sealyun.com
interface 默认值为"eth.* en.* em.*"
ipip 是否开起ipip模式默认值为true
kubeadm-config kubeadm-config.yaml模板文件地址
lvscare-image lvscare镜像名默认值为"fanux/lvscare"
lvscare-tag lvscare镜像标签默认值为“latest”
mtu ipip模式的mtu值默认值为"1440"
network 指定安装安装的CNI组件默认值为"calico"
pk SSH私钥文件路径默认值为"/root/.ssh/id_rsa"
pk-passwd SSH私钥密码
podcidr POD的IP地址网络段默认值为"100.64.0.0/10"
repo 指定安装时镜像拉取的镜像注册仓库地址默认值为"k8s.gcr.io"
svccidr k8s service的IP地址网络段默认值为"10.96.0.0/12"
user ssh用户名默认值为"root"
vip 虚拟IP地址默认值为"10.103.97.2"
vlog 设置kubeadm的日志级别
without-cni 如果设置为true，则不安装CNI组件
config 集群配置文件路径（全局配置参数）默认为$HOME/.sealos/config.yaml
info 指定安装时的日志输出级别 true为Info级别，false为Debug级别

新版

# Run a single node kubernetes
$ sealos run labring/kubernetes:v1.24.0 labring/calico:v3.22.1

# Run a HA kubernetes cluster
$ sealos run labring/kubernetes:v1.24.0 labring/calico:v3.22.1 \
     --masters 192.168.64.2,192.168.64.22,192.168.64.20 \
     --nodes 192.168.64.21,192.168.64.19 -p [your-ssh-passwd]

# Add masters or nodes
$ sealos add --masters 192.168.64.20 --nodes 192.168.64.21,192.168.64.22

# Delete your cluster
$ sealos reset

sealos部署k8s集群

一、 在线安装k8s集群

1、准备服务器

2、服务器配置

3、二进制安装sealos

4、自定义安装配置

5、开始安装集群

6、查看集群

报错解决

问题1：function “semverCompare” not defined

问题2：exec auth.sh failed

解决方式

二、离线安装k8s集群

下载离线包

安装k8s集群

老版

参数说明：

新版

悦读

一、在线安装k8s集群