[CKA] KodeKloud - Cluster Upgrade Process

 

 

안녕하세요, 쯀리입니다.

오늘은 Cluster Upgrade 방법에 대해 알아보겠습니다. 

 

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

Upgrading kubeadm clusters

 

---

 

Cluster Upgrade 방법

Kubernetes 클러스터 업그레이드는 여러 단계로 이루어지며, 각 단계는 신중하게 계획되고 수행되어야 합니다. 여기서는 주요 클러스터 관리 도구인 kubeadm을 사용하여 클러스터를 업그레이드하는 방법을 설명하겠습니다.

Kubernetes 클러스터 업그레이드 단계

1. 클러스터 상태 확인
2. 마스터 노드 업그레이드
3. 워커 노드 업그레이드
4. 애드온 및 기타 구성요소 업그레이드

 

 

 

---

Quiz

1. This lab tests your skills on upgrading a kubernetes cluster. We have a production cluster with applications running on it. Let us explore the setup first.

What is the current version of the cluster?

controlplane ~ ➜  kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.0", GitCommit:"855e7c48de7388eb330da0f8d9d2394ee818fb8d", GitTreeState:"clean", BuildDate:"2023-08-15T10:20:15Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}

 

v1.28 

2. How many nodes are part of this cluster? Including controlplane and worker nodes

controlplane ~ ➜  k get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   31m   v1.28.0
node01         Ready    <none>          31m   v1.28.0

 

3. How many nodes can host workloads in this cluster? Inspect the applications and taints set on the nodes.

controlplane ~ ✖ k describe node controlplane | grep Taint
Taints:             <none>

controlplane ~ ➜  k describe node node01 | grep Taint
Taints:             <none>

두 node 모두 Taint가 none으로 설정되어있기 때문에 모두 workload를 host할수 있습니다. 

4. How many applications are hosted on the cluster? Count the number of deployments in the default namespace.

controlplane ~ ➜  k get deployment
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
blue   5/5     5            5           6m35s

 

5. What nodes are the pods hosted on?

controlplane ~ ➜  k get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
blue-667bf6b9f9-5cq89   1/1     Running   0          7m17s   10.244.0.5   controlplane   <none>           <none>
blue-667bf6b9f9-8jr9k   1/1     Running   0          7m17s   10.244.1.3   node01         <none>           <none>
blue-667bf6b9f9-bqbhh   1/1     Running   0          7m17s   10.244.0.4   controlplane   <none>           <none>
blue-667bf6b9f9-ns296   1/1     Running   0          7m17s   10.244.1.2   node01         <none>           <none>
blue-667bf6b9f9-vfvcd   1/1     Running   0          7m17s   10.244.1.4   node01         <none>           <none>

controlplane과 node01 모두 pod가 사용중입니다. 

6. You are tasked to upgrade the cluster. Users accessing the applications must not be impacted, and you cannot provision new VMs. What strategy would you use to upgrade the cluster? 

Upgrade one node at a time while moving the workloads to the other

하나의 node씩 작업하는 것이 좋습니다.

7.  What is the latest version available for an upgrade with the current version of the kubeadm tool installed?
Use the kubeadm tool

 

controlplane ~ ➜  sudo kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.28.0
[upgrade/versions] kubeadm version: v1.28.0
vI0711 10:46:58.765629   14154 version.go:256] remote version is much newer: v1.30.2; falling back to: stable-1.28
[upgrade/versions] Target version: v1.28.11
[upgrade/versions] Latest version in the v1.28 series: v1.28.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     2 x v1.28.0   v1.28.11

Upgrade to the latest version in the v1.28 series:

COMPONENT                 CURRENT   TARGET
kube-apiserver            v1.28.0   v1.28.11
kube-controller-manager   v1.28.0   v1.28.11
kube-scheduler            v1.28.0   v1.28.11
kube-proxy                v1.28.0   v1.28.11
CoreDNS                   v1.10.1   v1.10.1
etcd                      3.5.9-0   3.5.9-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.28.11

Note: Before you can perform this upgrade, you have to update kubeadm to v1.28.11.

v1.28.11로 새로 업그레이드 가능합니다!

8. We will be upgrading the controlplane node first. Drain the controlplane node of workloads and mark it UnSchedulable

controlplane ~ ➜  k drain controlplane --ignore-daemonsets
node/controlplane cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-gzrzh, kube-system/kube-proxy-kfxfn
evicting pod kube-system/coredns-5dd5756b68-pv5jw
evicting pod default/blue-667bf6b9f9-bqbhh
evicting pod default/blue-667bf6b9f9-5cq89
evicting pod kube-system/coredns-5dd5756b68-gdbmm
pod/blue-667bf6b9f9-bqbhh evicted
pod/blue-667bf6b9f9-5cq89 evicted
pod/coredns-5dd5756b68-pv5jw evicted
pod/coredns-5dd5756b68-gdbmm evicted
node/controlplane drained

 

9. Upgrade the controlplane components to exact version v1.29.0

Upgrade the kubeadm tool (if not already), then the controlplane components, and finally the kubelet. Practice referring to the Kubernetes documentation page.

vim /etc/apt/sources.list.d/kubernetes.list

## 뒤에를 1.29버전으로 변경
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /


root@controlplane:~# apt update
root@controlplane:~# apt-cache madison kubeadm
root@controlplane:~# apt-get install kubeadm=1.29.0-1.1
root@controlplane:~# kubeadm upgrade apply v1.29.0

## kubelet 업그레이드
## 워커 노드의 kubelet을 업그레이드합니다.
root@controlplane:~# apt-get install kubelet=1.29.0-1.1
root@controlplane:~# systemctl daemon-reload
root@controlplane:~# systemctl restart kubelet
root@controlplane:~# kubectl uncordon controlplane

 

controlplane ~ ➜  k get nodes
NAME           STATUS                     ROLES           AGE   VERSION
controlplane   Ready,SchedulingDisabled   control-plane   52m   v1.29.0
node01         Ready                      <none>          51m   v1.28.0

 

 

10. Mark the controlplane node as "Schedulable" again

controlplane ~ ✖ k uncordon controlplane
node/controlplane uncordoned

controlplane ~ ➜  k get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   53m   v1.29.0
node01         Ready    <none>          53m   v1.28.0

 

11. Next is the worker node. Drain the worker node of the workloads and mark it UnSchedulable

worker node인 node01을 중지시키겠습니다.

controlplane ~ ➜  k drain node01 --ignore-daemonsets
node/node01 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-flannel/kube-flannel-ds-f58kt, kube-system/kube-proxy-b77rl
evicting pod kube-system/coredns-76f75df574-ffpcs
evicting pod default/blue-667bf6b9f9-fhzdb
evicting pod default/blue-667bf6b9f9-vfvcd
evicting pod default/blue-667bf6b9f9-4lfcj
evicting pod default/blue-667bf6b9f9-8jr9k
evicting pod kube-system/coredns-76f75df574-fcccn
evicting pod default/blue-667bf6b9f9-ns296
pod/blue-667bf6b9f9-4lfcj evicted
pod/blue-667bf6b9f9-vfvcd evicted
pod/blue-667bf6b9f9-ns296 evicted
pod/blue-667bf6b9f9-fhzdb evicted
I0711 11:01:33.481982   22324 request.go:697] Waited for 1.004311864s due to client-side throttling, not priority and fairness, request: GET:https://controlplane:6443/api/v1/namespaces/default/pods/blue-667bf6b9f9-8jr9k
pod/blue-667bf6b9f9-8jr9k evicted
pod/coredns-76f75df574-ffpcs evicted
pod/coredns-76f75df574-fcccn evicted
node/node01 drained

 

12. Upgrade the worker node to the exact version v1.29.0

위에 실행한 방법대로 그대로 실행해주겠습니다. (base는 controlplane이기 때문에 ssh로 node01로 접속하겠습니다. )

controlplane ~ ✖ ssh node01

root@node01 ~ ➜

    1  vim /etc/apt/sources.list.d/kubernetes.list
    2  apt update
    4  apt-cache madison kubeadm
    5  apt-get install kubeadm=1.29.0-1.1
    6  kubeadm upgrade node
    7  apt-get install kubelet=1.29.0-1.1
    8  systemctl daemon-reload
    9  systemctl restart kubelet
   10  history

 

13. Remove the restriction and mark the worker node as schedulable again.

controlplane ~ ✖ k get nodes 
NAME           STATUS                     ROLES           AGE   VERSION
controlplane   Ready                      control-plane   61m   v1.29.0
node01         Ready,SchedulingDisabled   <none>          61m   v1.29.0

controlplane ~ ➜  k uncordon node01
node/node01 uncordoned

controlplane ~ ➜  k get nodes
NAME           STATUS   ROLES           AGE   VERSION
controlplane   Ready    control-plane   61m   v1.29.0
node01         Ready    <none>          61m   v1.29.0

controlplane ~ ➜

---

오늘은 Cluster Upgrade 방법들에 대해 알아보았는데 정말 어렵져..?

다음시간에는 Backup and Restore Methods 이부분을 공부해보겠습니다 !

 

---

참조

※ Udemy Labs - Certified Kubernetes Administrator with Practice Tests