Loading... ### 系统环境: ``` Ubuntu 22.04 LTS containerd 1.6.20 k8s-mater:192.168.60.160 k8s-node1:192.168.60.161 k8s-node2:192.168.60.162 ``` ### 主机基础优化(所有节点): ``` cat /etc/sysctl.conf net.ipv4.ip_forward=1 vm.max_map_count=262144 kernel.pid_max=4194303 fs.file-max=1000000 net.ipv4.tcp_max_tw_buckets=6000 net.netfilter.nf_conntrack_max=2097152 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 vm.swappiness=0 ``` net.ipv4.ip\_forward=1: 启用IP转发功能,用于将接收到的网络数据包转发到另一个网络接口,有助于实现路由、防火墙等网络功能。 vm.max\_map\_count=262144:设置进程可以拥有的内存区域数量上限,通常应该大于或等于 Elasticsearch 和 Solr 需要的最低数量。 kernel.pid\_max=4194303:设置进程 ID 的上限,即最大 PID。这个值越大,允许的进程数也就越多。 fs.file-max=1000000:设置系统可以同时打开的文件句柄数上限,这个值的大小决定了系统能够同时运行的应用程序数量。 net.ipv4.tcp\_max\_tw\_buckets=6000:设置本地核心分配的 TIME\_WAIT sockets 的最大数量,这决定了可以生成的TCP连接数量。 net.netfilter.nf\_conntrack\_max=2097152:设置Netfilter连接跟踪状态表大小的上限,跟踪表缓存容量的大小设置会影响到连接跟踪的引擎,上限防止服务器中规模较大的回话跟踪出现阻塞网络流量情况而设置。 net.bridge.bridge-nf-call-ip6tables = 1:如果启用了 net.ipv4.ip\_forward 参数,需要将该参数设置为1。这样可以确保内核能够在 Linux 虚拟网桥桥接 IPv6 数据流时自动应用 iptables 规则。 net.bridge.bridge-nf-call-iptables = 1:开启系统 IPVS 时,如果启用了 `net.ipv4.ip_forward`,还需将该参数设置为1,以确保内核在 Linux 虚拟网桥桥接 IPv4 数据流时自动应用 iptables 规则。 vm.swappiness=0:控制系统进行分页交换(swap)的倾向,值为0表示操作系统只在极端情况下才会进行交换(尽可能少)。 ### 内核模块开机挂载(所有节点): ``` cat <<EOF >/etc/modules-load.d/modules.conf ip_vs ip_vs_lc ip_vs_lblc ip_vs_lblcr ip_vs_rr ip_vs_wrr ip_vs_sh ip_vs_dh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp ip_vs_sh ip_tables ip_set ipt_set ipt_rpfilter ipt_REJECT ipip xt_set br_netfilter nf_conntrack overlay EOF ``` ### 关闭swap(所有节点): 临时关闭: ``` sudo swapoff -a ``` 永久关闭swap: ``` 编辑/etc/fstab文件 root@k8s-master:~# cat /etc/fstab # /etc/fstab: static file system information. :# # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point> <type> <options> <dump> <pass> # / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation /dev/disk/by-id/dm-uuid-LVM-B3JMt5tYZ6wr0jJ9KKyrqy9Wrkh0YnhH55zotzNSKjdq7mjFQ739NvDv5HIV5UP8 / ext4 defaults 0 1 # /boot was on /dev/sda2 during curtin installation /dev/disk/by-uuid/39a04650-f093-4cde-ae62-1a6c36fa2b63 /boot ext4 defaults 0 1 #/swap.img none swap sw 0 0 ``` 注释掉所有swap分区的行或者删除swap分区的行 重启系统 ``` sudo reboot ``` 重启后验证内核模块与内存参数: ``` lsmod | grep br_netfilter br_netfilter 32768 0 bridge 307200 1 br_netfilter sysctl -a | grep bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables = 1 ``` 安装containerd(所有节点): ``` wget https://shackles.cn/Software/runtime-docker20.10.19-containerd1.6.20-binary-install.tar.gz tar xvf runtime-docker20.10.19-containerd1.6.20-binary-install.tar.gz bash runtime-install.sh containerd ``` ### 更换软件仓库镜像: ``` cat <<EOF >/etc/apt/sources.list # 默认注释了源码镜像以提高 apt update 速度,如有需要可自行取消注释 deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-updates main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse # deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-backports main restricted universe multiverse deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ jammy-security main restricted universe multiverse EOF ``` ### 各节点安装kubeadm、 kubectl、 kubelet: ``` apt-get update && apt-get install -y apt-transport-https -y curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add - cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main EOF apt-get update && apt-cache madison kubeadm apt-get install -y kubeadm=1.27.2-00 kubectl=1.27.2-00 kubelet=1.27.2-00 ``` ### 下载kubenetes镜像: ``` kubeadm config images list --kubernetes-version v1.27.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controllermanager:v1.27.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.2 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.7-0 registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 #master节点所需image cat <<EOF >/root/images-master-down.sh #!/bin/bash nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.27.2 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controllermanager:v1.27.2 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.27.2 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.7-0 EOF #node节点所需image cat <<EOF >/root/images-node-down.sh #!/bin/bash nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.2 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 nerdctl pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 EOF ``` 使用配置文件进行初始化master: ``` kubeadm config print init-defaults > kubeadm-init.yaml vim kubeadm-init.yaml apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s #加入集群需要的token有效时常默认24小时 usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.60.160 # master主机的IP地址也是API Server的监听地址 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock #可ls此文件查看是否存在 imagePullPolicy: IfNotPresent name: k8s-master #当前主机名 taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki #证书路径 clusterName: kubernetes #controlPlaneEndpoint: 192.168.60.160:6443 #公有云中负载均衡器的内网地址 controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers #修改为国内image仓库 kind: ClusterConfiguration kubernetesVersion: 1.27.2 #和当前版本对应 networking: dnsDomain: cluster.local podSubnet: 10.200.0.0/16 #Pod的IP段 serviceSubnet: 10.100.0.0/16 #SVC的IP段 scheduler: {} --- #指定kubelet使⽤systemd kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 cgroupDriver: systemd --- #指定KubeProxy使⽤ipvs apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: ipvs ``` 初始化: ``` kubeadm init --config kubeadm-init.yaml ``` 初始化成功后会打印如下信息: ``` Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.60.160:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:4e645a1f0d5d377f79d87139232e35e5972ebe86dbca1df808a54e818be3ec58 ``` 根据提示执行指令: ``` mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` 这三条命令的作用是将 Kubernetes 集群的 admin.conf 配置文件拷贝到当前用户的 \$HOME/.kube/config 文件中,并修改其拥有者为当前用户,以便可以通过 kubectl 命令管理 Kubernetes 集群。 ### 添加Node节点: ``` kubeadm join 192.168.60.160:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:4e645a1f0d5d377f79d87139232e35e5972ebe86dbca1df808a54e818be3ec58 ``` ### 检查node节点是否加入成功: ``` kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master NotReady control-plane 20h v1.27.2 k8s-node1 NotReady <none> 16h v1.27.2 k8s-node2 NotReady <none> 16h v1.27.2 如上图所示nodes又两台已经加入成功,STATUS为NotReady是因为没有配置CNI。 ``` ### 部署⽹络组件flannel: ``` cat <<EOF > kube-flannel.yml apiVersion: v1 kind: Namespace metadata: labels: k8s-app: flannel pod-security.kubernetes.io/enforce: privileged name: kube-flannel --- apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: flannel name: flannel namespace: kube-flannel --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: flannel name: flannel rules: - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - get - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch - apiGroups: - networking.k8s.io resources: - clustercidrs verbs: - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: flannel name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-flannel --- apiVersion: v1 data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.200.0.0/16", "Backend": { "Type": "vxlan" } } kind: ConfigMap metadata: labels: app: flannel k8s-app: flannel tier: node name: kube-flannel-cfg namespace: kube-flannel --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app: flannel k8s-app: flannel tier: node name: kube-flannel-ds namespace: kube-flannel spec: selector: matchLabels: app: flannel k8s-app: flannel template: metadata: labels: app: flannel k8s-app: flannel tier: node spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/os operator: In values: - linux containers: - args: - --ip-masq - --kube-subnet-mgr command: - /opt/bin/flanneld env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: EVENT_QUEUE_DEPTH value: "5000" image: docker.io/flannel/flannel:v0.22.0 name: kube-flannel resources: requests: cpu: 100m memory: 50Mi securityContext: capabilities: add: - NET_ADMIN - NET_RAW privileged: false volumeMounts: - mountPath: /run/flannel name: run - mountPath: /etc/kube-flannel/ name: flannel-cfg - mountPath: /run/xtables.lock name: xtables-lock hostNetwork: true initContainers: - args: - -f - /flannel - /opt/cni/bin/flannel command: - cp image: docker.io/flannel/flannel-cni-plugin:v1.1.2 name: install-cni-plugin volumeMounts: - mountPath: /opt/cni/bin name: cni-plugin - args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist command: - cp image: docker.io/flannel/flannel:v0.22.0 name: install-cni volumeMounts: - mountPath: /etc/cni/net.d name: cni - mountPath: /etc/kube-flannel/ name: flannel-cfg priorityClassName: system-node-critical serviceAccountName: flannel tolerations: - effect: NoSchedule operator: Exists volumes: - hostPath: path: /run/flannel name: run - hostPath: path: /opt/cni/bin name: cni-plugin - hostPath: path: /etc/cni/net.d name: cni - configMap: name: kube-flannel-cfg name: flannel-cfg - hostPath: path: /run/xtables.lock type: FileOrCreate name: xtables-lock EOF 配置中"Network": "10.200.0.0/16",要和初始化时指定的podSubnet对应。 kubectl apply -f kube-flannel.yml namespace/kube-flannel created serviceaccount/flannel created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created ``` #### 验证flannel是否部署成功: ``` root@k8s-master:~# kubectl get pods -n kube-flannel NAME READY STATUS RESTARTS AGE kube-flannel-ds-57hbz 1/1 Running 0 15h kube-flannel-ds-hg48m 1/1 Running 0 15h kube-flannel-ds-w5dsj 1/1 Running 0 15h ``` ### 验证node节点状态: ``` kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready control-plane 20h v1.27.2 k8s-node1 Ready <none> 16h v1.27.2 k8s-node2 Ready <none> 16h v1.27.2 ``` ### 后期添加master节点: ``` kubeadm init phase upload-certs --upload-certs I0414 16:21:59.283672 14673 version.go:256] remote version is much newer: v1.27.0; falling back to: stable-1.26 [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: e4226754e61e5cbcec412db6693597e2489ccf62187137357bd40a50b6c9b313 root@k8s-master2:~# kubeadm join 192.168.60.160:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:5e36ce948f5f2ef7af33d61664d6572a4818239c4ec593d4fb0ee66eda17dfea \ --control-plane --certificate-key e4226754e61e5cbcec412db6693597e2489ccf62187137357bd40a50b6c9b313 ``` #### 如遇后期问题 ```shell 1、work节点无法执行kubectl命令 [root@k8s-work01 ~]# kubectl get node The connection to the server localhost:8080 was refused - did you specify the right host or port? 解决方法: (1)、work节点执行如下命令 echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile, source ~/.bash_profile mkdir -p ~/.kube (2)master节点执行如下命令 scp /etc/kubernetes/admin.conf root@k8s-work:/etc/kubernetes/ scp ~/.kube/config root@k8s-work01:~/.kube/config ``` 最后修改:2025 年 04 月 08 日 © 允许规范转载 打赏 赞赏作者 支付宝微信 赞 2 如果觉得我的文章对你有用,请随意赞赏