kubelet服务启动失败问题处理

1.问题重现

[root@leoheng-k8s ~]# systemctl enable kubelet && systemctl start kubelet
[root@leoheng-k8s ~]# systemctl status  kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-01-28 09:56:54 CST; 7s ago
     Docs: https://kubernetes.io/docs/
  Process: 2717 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
 Main PID: 2717 (code=exited, status=255)

Jan 28 09:56:54 leoheng-k8s systemd[1]: Unit kubelet.service entered failed state.
Jan 28 09:56:54 leoheng-k8s systemd[1]: kubelet.service failed.

2.问题处理

1.关闭selinux、firewalld，并且把cgroup设置与docker一样

2.查看官方文档

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. This crashloop is expected and normal, please proceed with the next step and the kubelet will start running normally.

3.执行k8s的初始化

[root@leoheng-k8s ~]# kubeadm init
[init] Using Kubernetes version: v1.20.2
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.2. Latest validated version: 19.03
	[WARNING Hostname]: hostname "leoheng-k8s" could not be reached
	[WARNING Hostname]: hostname "leoheng-k8s": lookup leoheng-k8s on 100.100.2.138:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local leoheng-k8s] and IPs [10.96.0.1 172.18.192.80]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [leoheng-k8s localhost] and IPs [172.18.192.80 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [leoheng-k8s localhost] and IPs [172.18.192.80 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.002755 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node leoheng-k8s as control-plane by adding the labels "node-role.kubernetes.io/master=''" and "node-role.kubernetes.io/control-plane='' (deprecated)"
[mark-control-plane] Marking the node leoheng-k8s as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 1mxqc0.zwbjh8m8g5d35foe
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.18.192.80:6443 --token 1mxqc0.zwbjh8m8g5d35foe \
    --discovery-token-ca-cert-hash sha256:09949a2800da77bd71d046080fe5a75662472a32d0070305ce1fc7457a642b2d 

4.重新启动kubectl服务

[root@leoheng-k8s ~]# systemctl status  kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Thu 2021-01-28 09:58:49 CST; 2min 8s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 4088 (kubelet)
    Tasks: 13
   Memory: 37.8M
   CGroup: /system.slice/kubelet.service
           └─4088 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/...

Jan 28 10:00:34 leoheng-k8s kubelet[4088]: W0128 10:00:34.351824    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:36 leoheng-k8s kubelet[4088]: E0128 10:00:36.029903    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:39 leoheng-k8s kubelet[4088]: W0128 10:00:39.352002    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:41 leoheng-k8s kubelet[4088]: E0128 10:00:41.044753    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:44 leoheng-k8s kubelet[4088]: W0128 10:00:44.352167    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:46 leoheng-k8s kubelet[4088]: E0128 10:00:46.060000    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:49 leoheng-k8s kubelet[4088]: W0128 10:00:49.352345    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:51 leoheng-k8s kubelet[4088]: E0128 10:00:51.075055    4088 kubelet.go:2163] Container runtim...lized
Jan 28 10:00:54 leoheng-k8s kubelet[4088]: W0128 10:00:54.352524    4088 cni.go:239] Unable to update cni ...net.d
Jan 28 10:00:56 leoheng-k8s kubelet[4088]: E0128 10:00:56.090169    4088 kubelet.go:2163] Container runtim...lized
Hint: Some lines were ellipsized, use -l to show in full.
[root@leoheng-k8s ~]# 

• 本文作者： Leoheng
• 本文链接： http://www.leoheng.com/2021/01/31/kubelet服务启动失败问题处理/
• 版权声明： Design by leoheng