kubelet이 실행되지 않을때

November 05, 2019 (Last Modified: November 05, 2019)

Page content

mini1 리붓 후 ghost 접속이 안됨.

docker를 직접 실행시키는 wordpress는 정상적으로 실행

그래서 kubectl get svc 명령을 치니 접속이 안된다고.

$ ps -ef |grep kube
cychong   7461  2486  0 23:26 pts/0    00:00:00 grep --color=auto kube
$ service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-11-04 23:26:47 KST; 5s ago
     Docs: https://kubernetes.io/docs/home/
  Process: 7664 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255
 Main PID: 7664 (code=exited, status=255)

$ journalctl -xeu kubelet
Nov 04 23:28:32 mini1 kubelet[8695]: I1104 23:28:32.418274    8695 server.go:773] Client rotation is on, will bootstrap in background
Nov 04 23:28:32 mini1 kubelet[8695]: I1104 23:28:32.427223    8695 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-clien
Nov 04 23:28:32 mini1 kubelet[8695]: I1104 23:28:32.580296    8695 server.go:644] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulti
Nov 04 23:28:32 mini1 kubelet[8695]: F1104 23:28:32.580915    8695 server.go:271] failed to run Kubelet: running with swap on is not supported, please disa
Nov 04 23:28:32 mini1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Nov 04 23:28:32 mini1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Nov 04 23:28:42 mini1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Nov 04 23:28:42 mini1 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 30.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit kubelet.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Nov 04 23:28:42 mini1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished shutting down.
Nov 04 23:28:42 mini1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit kubelet.service has finished starting up.
--
-- The start-up result is RESULT.
Nov 04 23:28:42 mini1 kubelet[8795]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's
Nov 04 23:28:42 mini1 kubelet[8795]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --
Nov 04 23:28:42 mini1 kubelet[8795]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's
Nov 04 23:28:42 mini1 kubelet[8795]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --
Nov 04 23:28:42 mini1 kubelet[8795]: I1104 23:28:42.907863    8795 server.go:410] Version: v1.16.1
Nov 04 23:28:42 mini1 kubelet[8795]: I1104 23:28:42.908251    8795 plugins.go:100] No cloud provider specified.
Nov 04 23:28:42 mini1 kubelet[8795]: I1104 23:28:42.908283    8795 server.go:773] Client rotation is on, will bootstrap in background
Nov 04 23:28:42 mini1 kubelet[8795]: I1104 23:28:42.917653    8795 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-clien
Nov 04 23:28:43 mini1 kubelet[8795]: I1104 23:28:43.073234    8795 server.go:644] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulti
Nov 04 23:28:43 mini1 kubelet[8795]: F1104 23:28:43.073886    8795 server.go:271] failed to run Kubelet: running with swap on is not supported, please disa
Nov 04 23:28:43 mini1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Nov 04 23:28:43 mini1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
lines 1947-1986/1986 (END)

찾아보니 docker가 정상적으로 실행 중인지 확인해 보라고.

하지만 정상적으로 동작 중

$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-11-04 23:25:40 KST; 7min ago
     Docs: https://docs.docker.com
 Main PID: 5285 (dockerd)
    Tasks: 59
   CGroup: /system.slice/docker.service
           ├─5285 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
           ├─5586 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.17.0.3 -container-port 80
           └─5598 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8181 -container-ip 172.17.0.4 -container-port 80

Nov 04 23:25:35 mini1 dockerd[5285]: time="2019-11-04T23:25:35.378057010+09:00" level=warning msg="Your kernel does not support swap memory limit"
Nov 04 23:25:35 mini1 dockerd[5285]: time="2019-11-04T23:25:35.378125944+09:00" level=warning msg="Your kernel does not support cgroup rt period"
Nov 04 23:25:35 mini1 dockerd[5285]: time="2019-11-04T23:25:35.378148389+09:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Nov 04 23:25:35 mini1 dockerd[5285]: time="2019-11-04T23:25:35.378414322+09:00" level=info msg="Loading containers: start."
Nov 04 23:25:36 mini1 dockerd[5285]: time="2019-11-04T23:25:36.163613079+09:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172
Nov 04 23:25:39 mini1 dockerd[5285]: time="2019-11-04T23:25:39.843337707+09:00" level=info msg="Loading containers: done."
Nov 04 23:25:40 mini1 dockerd[5285]: time="2019-11-04T23:25:40.046412649+09:00" level=info msg="Docker daemon" commit=9013bf583a graphdriver(s)=overlay2 ve
Nov 04 23:25:40 mini1 dockerd[5285]: time="2019-11-04T23:25:40.047017959+09:00" level=info msg="Daemon has completed initialization"
Nov 04 23:25:40 mini1 dockerd[5285]: time="2019-11-04T23:25:40.109258901+09:00" level=info msg="API listen on /var/run/docker.sock"
Nov 04 23:25:40 mini1 systemd[1]: Started Docker Application Container Engine.

범인은 아니 문제에 대한 설명은 에러 메시지에 있을 확률이 가장 높으니 위 systemctl status kubelet의 결과를 찬찬히 살펴보니 swap에 대한 내용이 눈에 띈다.

그래서 바로 해결 시도.

$ sudo swapoff -a
[sudo] password for cychong:

cychong@mini1:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Mon 2019-11-04 23:48:43 KST; 2s ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 18121 (kubelet)
    Tasks: 11 (limit: 4306)
   CGroup: /system.slice/kubelet.service
           └─18121 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/l

Nov 04 23:48:43 mini1 kubelet[18121]: I1104 23:48:43.741976   18121 remote_image.go:50] parsed scheme: ""
Nov 04 23:48:43 mini1 kubelet[18121]: I1104 23:48:43.742008   18121 remote_image.go:50] scheme "" not registered, fallback to default scheme
Nov 04 23:48:43 mini1 kubelet[18121]: I1104 23:48:43.742038   18121 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock
Nov 04 23:48:43 mini1 kubelet[18121]: I1104 23:48:43.742058   18121 clientconn.go:577] ClientConn switching balancer to "pick_first"
Nov 04 23:48:44 mini1 kubelet[18121]: E1104 23:48:44.424387   18121 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node
Nov 04 23:48:44 mini1 kubelet[18121]: E1104 23:48:44.428655   18121 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Serv
Nov 04 23:48:44 mini1 kubelet[18121]: E1104 23:48:44.429959   18121 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list
Nov 04 23:48:45 mini1 kubelet[18121]: E1104 23:48:45.425295   18121 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node
Nov 04 23:48:45 mini1 kubelet[18121]: E1104 23:48:45.429648   18121 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Serv
Nov 04 23:48:45 mini1 kubelet[18121]: E1104 23:48:45.430623   18121 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list

우연치 않게 회사에서도 비슷한 문제가 있었는데 그것도 swap이 원인이었다는…