Upgrade kubernetes to 1.16.1
Page content
cychong@mini1:~/work/ghost-with-helm-x$ sudo apt update
[sudo] password for cychong:
Ign:2 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:4 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:5 http://dl.google.com/linux/chrome/deb stable Release
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Get:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata [295 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main DEP-11 48x48 Icons [73.8 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main DEP-11 64x64 Icons [147 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 DEP-11 Metadata [254 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/universe DEP-11 48x48 Icons [197 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/universe DEP-11 64x64 Icons [453 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2468 B]
Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 DEP-11 Metadata [7916 B]
Get:18 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 Packages [526 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-security/main Translation-en [176 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [38.5 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-security/main DEP-11 48x48 Icons [17.6 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-security/main DEP-11 64x64 Icons [41.5 kB]
Get:23 http://archive.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [611 kB]
Get:24 http://archive.ubuntu.com/ubuntu bionic-security/universe Translation-en [203 kB]
Get:25 http://archive.ubuntu.com/ubuntu bionic-security/universe amd64 DEP-11 Metadata [42.2 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic-security/universe DEP-11 48x48 Icons [16.4 kB]
Get:27 http://archive.ubuntu.com/ubuntu bionic-security/universe DEP-11 64x64 Icons [111 kB]
Get:28 http://archive.ubuntu.com/ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2464 B]
Fetched 3467 kB in 27s (128 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
41 packages can be upgraded. Run 'apt list --upgradable' to see them.
cychong@mini1:~/work/ghost-with-helm-x$ sudo apt-cache policy kubeadm
kubeadm:
Installed: 1.15.3-00
Candidate: 1.16.1-00
cychong@mini1:~/work/ghost-with-helm-x$ sudo apt-get install -y kubeadm=1.16.1-00 && sudo apt-mark hold kubeadm
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
libllvm7
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
kubeadm
1 upgraded, 0 newly installed, 0 to remove and 40 not upgraded.
Need to get 8764 kB of archives.
After this operation, 4062 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubeadm amd64 1.16.1-00 [8764 kB]
Fetched 8764 kB in 6s (1489 kB/s)
(Reading database ... 237550 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.16.1-00_amd64.deb ...
Unpacking kubeadm (1.16.1-00) over (1.15.3-00) ...
Setting up kubeadm (1.16.1-00) ...
kubeadm set on hold.
cychong@mini1:~/work/ghost-with-helm-x$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:58:27Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
cychong@mini1:~/work/ghost-with-helm-x$ sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.15.3
[upgrade/versions] kubeadm version: v1.16.1
[upgrade/versions] Latest stable version: v1.16.1
[upgrade/versions] Latest version in the v1.15 series: v1.15.4
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.15.3 v1.15.4
Upgrade to the latest version in the v1.15 series:
COMPONENT CURRENT AVAILABLE
API Server v1.15.3 v1.15.4
Controller Manager v1.15.3 v1.15.4
Scheduler v1.15.3 v1.15.4
Kube Proxy v1.15.3 v1.15.4
CoreDNS 1.3.1 1.6.2
Etcd 3.3.10 3.3.10
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.15.4
_____________________________________________________________________
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT AVAILABLE
Kubelet 1 x v1.15.3 v1.16.1
Upgrade to the latest stable version:
COMPONENT CURRENT AVAILABLE
API Server v1.15.3 v1.16.1
Controller Manager v1.15.3 v1.16.1
Scheduler v1.15.3 v1.16.1
Kube Proxy v1.15.3 v1.16.1
CoreDNS 1.3.1 1.6.2
Etcd 3.3.10 3.3.15-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.16.1
_____________________________________________________________________
failed
cychong@mini1:~/work/ghost-with-helm-x$ sudo kubeadm upgrade apply v1.16.1
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.16.1"
[upgrade/versions] Cluster version: v1.15.3
[upgrade/versions] kubeadm version: v1.16.1
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 0 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[upgrade/prepull] Failed prepulled the images for the control plane components error: the prepull operation timed out
To see the stack trace of this error execute with --v=5 or higher
retry
cychong@mini1:~$ sudo kubeadm upgrade apply v1.16.1
[sudo] password for cychong:
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[upgrade] Making sure the cluster is healthy:
[upgrade/version] You have chosen to change the cluster version to "v1.16.1"
[upgrade/versions] Cluster version: v1.15.3
[upgrade/versions] kubeadm version: v1.16.1
[upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y
[upgrade/prepull] Will prepull images for components [kube-apiserver kube-controller-manager kube-scheduler etcd]
[upgrade/prepull] Prepulling image for component etcd.
[upgrade/prepull] Prepulling image for component kube-controller-manager.
[upgrade/prepull] Prepulling image for component kube-scheduler.
[upgrade/prepull] Prepulling image for component kube-apiserver.
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-apiserver
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-etcd
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-controller-manager
[apiclient] Found 1 Pods for label selector k8s-app=upgrade-prepull-kube-scheduler
[upgrade/prepull] Prepulled image for component kube-apiserver.
[upgrade/prepull] Prepulled image for component etcd.
[upgrade/prepull] Prepulled image for component kube-controller-manager.
[upgrade/prepull] Prepulled image for component kube-scheduler.
[upgrade/prepull] Successfully prepulled the images for all the control plane components
[upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.16.1"...
Static pod: kube-apiserver-mini1 hash: 868871559cc75dab75f106d4af342538
Static pod: kube-controller-manager-mini1 hash: 44f6b9cce90e81a472520a3fb9751d10
Static pod: kube-scheduler-mini1 hash: 7d5d3c0a6786e517a8973fa06754cb75
[upgrade/etcd] Upgrading to TLS for etcd
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-07-23-43-23/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: 71542aaa2829652ef14b22098a4b46aa
Static pod: etcd-mini1 hash: d96090bab45a5dababb3c3015960926b
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests306281752"
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-07-23-43-23/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-apiserver-mini1 hash: 868871559cc75dab75f106d4af342538
Static pod: kube-apiserver-mini1 hash: 01800dd11dfbda441372caf7cbf8aa39
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-07-23-43-23/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-controller-manager-mini1 hash: 44f6b9cce90e81a472520a3fb9751d10
Static pod: kube-controller-manager-mini1 hash: e12d193633dcf11f6095d89ee58c45a9
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2019-10-07-23-43-23/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s)
Static pod: kube-scheduler-mini1 hash: 7d5d3c0a6786e517a8973fa06754cb75
Static pod: kube-scheduler-mini1 hash: bf9014e67294b0df0bc373fd7024ced7
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.16.1". Enjoy!
[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
Upgrade calico from v3.8 to v3.9
kubectl apply -f https://docs.projectcalico.org/v3.9/manifests/calico.yaml
configmap/calico-config unchanged
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org unchanged
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org unchanged
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node configured
clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged
daemonset.apps/calico-node configured
serviceaccount/calico-node unchanged
deployment.apps/calico-kube-controllers configured
serviceaccount/calico-kube-controllers unchanged
앗. 서브넷 변경하는 걸 깜빡
cychong@mini1:~$ wget https://docs.projectcalico.org/v3.9/manifests/calico.yaml
--2019-10-07 23:49:13-- https://docs.projectcalico.org/v3.9/manifests/calico.yaml
Resolving docs.projectcalico.org (docs.projectcalico.org)... 104.248.78.23, 2604:a880:2:d0::21e9:b001
Connecting to docs.projectcalico.org (docs.projectcalico.org)|104.248.78.23|:443... connected.
HTTP request sent, awaiting response... v200 OK
Length: 20648 (20K) [application/x-yaml]
Saving to: ‘calico.yaml’
calico.yaml 0%[ ] 0 --.-KB/s icalico.yaml 100%[=============================================================>] 20.16K 123KB/s in 0.2s
c2019-10-07 23:49:14 (123 KB/s) - ‘calico.yaml’ saved [20648/20648]
Change CALICO_IPV4POOL_CIDR
- name: CALICO_IPV4POOL_CIDR
value: "10.201.0.0/24"
Upgrade kubelet and kubectl
cychong@mini1:~$ sudo apt-mark unhold kubelet kubectl && sudo apt-get update && sudo apt-get install -y kubelet=1.16.1-00 kubectl=1.16.1-00 && sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Ign:2 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:3 http://dl.google.com/linux/chrome/deb stable Release
Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:5 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:1 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Hit:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:9 http://archive.ubuntu.com/ubuntu bionic-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
libllvm7
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 39 not upgraded.
Need to get 29.9 MB of archives.
After this operation, 7179 kB of additional disk space will be used.
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.16.1-00 [9234 kB]
Get:2 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubelet amd64 1.16.1-00 [20.7 MB]
Fetched 29.9 MB in 6s (4899 kB/s)
(Reading database ... 237550 files and directories currently installed.)
Preparing to unpack .../kubectl_1.16.1-00_amd64.deb ...
Unpacking kubectl (1.16.1-00) over (1.15.3-00) ...
Preparing to unpack .../kubelet_1.16.1-00_amd64.deb ...
Unpacking kubelet (1.16.1-00) over (1.15.3-00) ...
Setting up kubelet (1.16.1-00) ...
Setting up kubectl (1.16.1-00) ...
kubelet set on hold.
kubectl set on hold.
cychong@mini1:~$ sudo systemctl restart kubelet
cychong@mini1:~$
Check the status
cychong@mini1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
mini1 Ready master 29d v1.16.1
Reference