> For the complete documentation index, see [llms.txt](https://kubernetes.youngwjung.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kubernetes.youngwjung.com/core-concepts/statefulset/lab.md).

# 실습

### StatefulSet

1. Service 생성 

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: web
     labels:
       app: httpd
   spec:
     ports:
     - port: 80
     clusterIP: None
     selector:
       app: httpd
   EOF
   ```
2. 생성된 Service 및 Endpoint 확인 &#x20;

   ```
   kubectl get svc,ep -l app=httpd
   ```
3. StatefulSet 생성&#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: httpd
     labels:
       app: httpd
   spec:
     serviceName: web
     replicas: 1
     selector:
       matchLabels:
         app: httpd
     template:
       metadata:
         labels:
           app: httpd
       spec:
         containers:
         - name: httpd
           image: httpd
   EOF
   ```
4. 생성된 StatefulSet 및 Pod 확인 &#x20;

   ```
   kubectl get sts,pod -l app=httpd
   ```
5. Endpoint 확인 &#x20;

   ```
   kubectl get ep web
   ```
6. Endpoint에 포함된 Pod의 상세정보 확인

   ```
   kubectl get ep web -o=jsonpath='{.subsets}' | jq
   ```
7. 생성된 Pod의 이름 및 IP 확인

   ```
   kubectl get pod -l app=httpd -o wide
   ```
8. Pod 생성&#x20;

   ```
   kubectl run curl --image=curlimages/curl --command -- sleep 3600
   ```
9. cURL 명령어로 기존에 생성한 Pod에 배포된 웹서버 호출&#x20;

   ```
   kubectl exec curl -- curl -s httpd-0.web
   ```
10. StatefulSet의 Replica 수를 3개로 수정&#x20;

    ```
    kubectl scale statefulset httpd --replicas=3
    ```
11. 생성된 Pod 확인&#x20;

    ```
    kubectl get pod -l app=httpd
    ```
12. cURL 명령어로 새로 생성된 Pod에 배포된 웹서버 호출

    ```
    {
        kubectl exec curl -- curl -s httpd-1.web
        kubectl exec curl -- curl -s httpd-2.web
    }
    ```
13. 첫번째로 생성된 Pod를 삭제&#x20;

    ```
    kubectl delete pod httpd-0
    ```
14. Pod가 다시 생성되는지 확인&#x20;

    ```
    kubectl get pod -l app=httpd
    ```
15. 생성된 리스소 삭제&#x20;

    ```
    {
        kubectl delete sts httpd
        kubectl delete svc web 
        kubectl delete pod curl
    }
    ```

### Persistent Volume

1. Persistent Volume 종류 확인 - <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes>
2. 현재 설정된 Storage Class 확인&#x20;

   ```
   kubectl get storageclass
   ```
3. NFS 서버 생성

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: nfs
     labels:
       app: nfs
   spec:
     containers:
     - name: nfs
       image: erichough/nfs-server
       ports:
       - containerPort: 2049
       volumeMounts:
       - name: data
         mountPath: /data
       - name: kernel-module
         mountPath: /lib/modules
         readOnly: true
       env:
       - name: NFS_EXPORT_0
         value: "/data/ *(rw,sync,no_root_squash,subtree_check)"
       securityContext:
         capabilities:
           add:
           - CAP_SYS_ADMIN
           - SYS_MODULE
     initContainers:
     - name: index
       image: busybox
       command: ["/bin/chmod","-R","1777", "/data"]
       volumeMounts:
       - name: data
         mountPath: /data
     volumes:
     - name: data
       emptyDir: {}
     - name: kernel-module
       hostPath:
         path: /lib/modules
   EOF
   ```
4. NFS 서버 생성 확인

   ```
   kubectl get pod -l app=nfs
   ```
5. NFS 서버에 HTML 파일 생성

   ```
   kubectl exec nfs -c nfs -- \
   bash -c 'echo "Greeting from NFS-powered NGINX" > /data/index.html'
   ```
6. HTML 파일이 정상적으로 생성되었는지 확인

   ```
   kubectl exec nfs -c nfs -- cat /data/index.html
   ```
7. PersistentVolume 생성&#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: my-pv
   spec:
     storageClassName: manual
     capacity:
       storage: 1Gi
     accessModes:
       - ReadWriteMany
     persistentVolumeReclaimPolicy: Retain
     nfs:
       path: /data
       server: $(kubectl get pod -l app=nfs -o=jsonpath='{.items[0].status.podIP}')
       readOnly: false
   EOF
   ```
8. 생성된 PersistentVolume 확인&#x20;

   ```
   kubectl get pv my-pv
   ```
9. 생성된 PVC가 있는지 확인&#x20;

   ```
   kubectl get pvc
   ```
10. PVC 생성 &#x20;

    ```
    cat <<EOF | kubectl create -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: my-pvc
    spec:
      accessModes:
      - ReadWriteMany
      resources:
         requests:
           storage: 200Mi
      storageClassName: manual
    EOF
    ```
11. 생성된 PVC 확인&#x20;

    ```
    kubectl get pvc my-pvc
    ```
12. PV 확인&#x20;

    ```
    kubectl get pv my-pv
    ```
13. 데모 애플리케이션 배포

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
            volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: my-pvc
    EOF
    ```
14. 위에서 배포한 애플리케이션의 Pod가 정상적으로 생성되었는지 확인

    ```
    kubectl get pod -l app=nginx
    ```
15. 배포된 NGINX 웹서버들이 동일한 페이지를 보여주는지 확인

    ```
    for pod in $(kubectl get pod -l app=nginx -o=jsonpath='{.items[*].metadata.name}')
    do
      kubectl exec $pod -- curl -s localhost
    done
    ```
16. &#x20;PVC 상태 확인&#x20;

    ```
    kubectl get pvc
    ```
17. 하나의 Pod에 접속해서 PV가 마운트된 경로에 파일 생성&#x20;

    ```
    kubectl exec deploy/nginx -- \
    bash -c  "echo helloworld > /usr/share/nginx/html/hello.txt"
    ```
18. NFS 서버에 접속해서 위에서 생성한 파일이 보이는지 확인&#x20;

    ```
    kubectl exec nfs -c nfs -- cat /data/hello.txt
    ```
19. 배포된 Pod들에서 위에서 생성한 파일이 보이는지 확인

    ```
    for pod in $(kubectl get pod -l app=nginx -o=jsonpath='{.items[*].metadata.name}')
    do
      kubectl exec $pod -- curl -s localhost/hello.txt
    done
    ```
20. 데모 애플리케이션 삭제 &#x20;

    ```
    kubectl delete deploy nginx
    ```
21. NFS 서버에 접속해서 파일이 삭제되었는지 확인&#x20;

    ```
    kubectl exec nfs -c nfs -- ls /data
    ```
22. PVC 상태 확인&#x20;

    ```
    kubectl get pvc my-pvc
    ```
23. 데모 애플리케이션 재배포&#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
            volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: my-pvc
    EOF
    ```
24. 이전에 생성했던 파일이 존재 하는지 확인&#x20;

    ```
    for pod in $(kubectl get pod -l app=nginx -o=jsonpath='{.items[*].metadata.name}')
    do
      kubectl exec $pod -- ls /usr/share/nginx/html
    done
    ```
25. PVC 삭제  시도

    ```
    kubectl delete pvc my-pvc
    ```
26. 위에서 실행한 명령어가 완료되지 않을 경우에 Ctrl+C를 입력해서 명령어 종료
27. PVC 상태 확인&#x20;

    ```
    kubectl get pvc my-pvc
    ```
28. PVC에 명시된 Finalizers 확인 - <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection>

    ```
    kubectl get pvc my-pvc -o=jsonpath='{.metadata.finalizers}{"\n"}'
    ```
29. 데모 애플리케이션 삭제 &#x20;

    ```
    kubectl delete deploy nginx
    ```
30. PVC가 삭제되었는지 확인

    ```
    kubectl get pvc my-pvc
    ```
31. PV 상태 확인&#x20;

    ```
    kubectl get pv my-pv
    ```
32. PVC 생성 &#x20;

    ```
    cat <<EOF | kubectl create -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: my-pvc
    spec:
      accessModes:
      - ReadWriteMany
      resources:
         requests:
           storage: 200Mi
      storageClassName: manual
    EOF
    ```
33. PVC 상태 확인&#x20;

    ```
    kubectl get pvc my-pvc
    ```
34. PV에 명시된 PVC 정보 확인

    ```
    kubectl get pv my-pv -o=jsonpath='{.spec.claimRef}' | jq
    ```
35. 새로 생성한 PVC 상세정보 확인

    ```
    kubectl get pvc my-pvc -o yaml
    ```
36. PV 수정 - `spec.claimRef` 삭제&#x20;

    ```
    kubectl patch pv my-pv \
    --type=json -p='[{"op": "remove", "path": "/spec/claimRef"}]'
    ```
37. PVC 상태 확인&#x20;

    ```
    kubectl get pvc my-pvc
    ```
38. PV에 명시된 PVC 정보 확인

    ```
    kubectl get pv my-pv -o=jsonpath='{.spec.claimRef}' | jq
    ```
39. PVC의 uid 확인

    ```
    kubectl get pvc my-pvc -o=jsonpath='{.metadata.uid}{"\n"}'
    ```
40. PV 삭제 시도

    ```
    kubectl delete pv my-pv 
    ```
41. 위에서 실행한 명령어가 완료되지 않을 경우에 Ctrl+C를 입력해서 명령어 종료
42. PV 상태 확인&#x20;

    ```
    kubectl get pv my-pv
    ```
43. PV에 명시된 Finalizers 확인 - <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storage-object-in-use-protection>

    ```
    kubectl get pv my-pv -o=jsonpath='{.metadata.finalizers}{"\n"}'
    ```
44. PVC 삭제

    ```
    kubectl delete pvc my-pvc 
    ```
45. PV가 삭제되었는지 확인

    ```
    kubectl get pv my-pv
    ```
46. NFS 서버에 접속해서 파일이 삭제되었는지 확인&#x20;

    ```
    kubectl exec nfs -c nfs -- ls /data
    ```
47. NFS 서버 삭제&#x20;

    ```
    kubectl delete pod nfs 
    ```

### Dynamic Volume Provisioning

1. 현재 설정된 StorageClass 확인&#x20;

   ```
   kubectl get storageclass
   ```
2. PVC 생성

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: my-pvc
   spec:
     accessModes:
     - ReadWriteMany
     resources:
        requests:
          storage: 200Mi
   EOF
   ```
3. PVC 상태 확인

   ```
   kubectl get pvc my-pvc
   ```
4. PVC 상태가 Pending인 이유 확인

   ```
   kubectl describe pvc my-pvc
   ```
5. 데모 애플리케이션 배포

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: nginx
     name: nginx
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx
           ports:
           - containerPort: 80
           volumeMounts:
           - name: data
             mountPath: /mnt
         volumes:
         - name: data
           persistentVolumeClaim:
             claimName: my-pvc
   EOF
   ```
6. 위에서 배포한 애플리케이션의 Pod가 정상적으로 생성되었는지 확인

   ```
   kubectl get pod -l app=nginx
   ```
7. Pod 상태가 Pending인 이유 확인

   ```
   kubectl describe pod -l app=nginx
   ```
8. PVC 상태가 Pending인 이유 확인

   ```
   kubectl describe pvc my-pvc
   ```
9. PVC 재생성

   ```
   cat <<EOF | kubectl replace --force -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: my-pvc
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
        requests:
          storage: 200Mi
   EOF
   ```
10. Pod 상태 확인 - *Running 상태로 될때까지 시간이 걸릴수도 있음*

    ```
    kubectl get pod -l app=nginx
    ```
11. PVC 상태 확인

    ```
    kubectl get pvc my-pvc
    ```
12. PVC에 명시된 PV가 생성되었는지 확인&#x20;

    ```
    kubectl get pv $(kubectl get pvc my-pvc -o=jsonpath='{.spec.volumeName}')
    ```
13. PVC에 명시한 볼륨 요청 크기 만큼 실제로 할당되었는지 확인

    ```
    diff <(kubectl get pvc my-pvc -o=jsonpath='{.spec}' | jq) \
    <(kubectl get pvc my-pvc -o=jsonpath='{.status}' | jq) -y
    ```
14. EBS (gp2)의 최소 볼륨 크기 확인 - <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html#solid-state-drives>
15. 데모 애플리케이션 삭제

    ```
    kubectl delete deploy nginx
    ```
16. PVC 삭제

    ```
    kubectl delete pvc my-pvc 
    ```
17. PV가 삭제되었는지 확인 - *삭제 될때까지 시간이 걸릴수도 있음*

    ```
    kubectl get pv
    ```
18. LimitRange 생성

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: LimitRange
    metadata:
      name: storagelimits
    spec:
      limits:
      - type: PersistentVolumeClaim
        max:
          storage: 16Ti
        min:
          storage: 1Gi
    EOF
    ```
19. LimitRagne 확인&#x20;

    ```
    kubectl describe ns default
    ```
20. PVC 생성 시도

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: my-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
         requests:
           storage: 200Mi
    EOF
    ```
21. PVC 생성

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: my-pvc
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
         requests:
           storage: 2Gi
    EOF
    ```
22. PVC 상태 확인

    ```
    kubectl get pvc my-pvc
    ```
23. 데모 애플리케이션 배포&#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
            volumeMounts:
            - name: data
              mountPath: /mnt
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: my-pvc
    EOF
    ```
24. Pod 상태 확인

    ```
    kubectl get pod -l app=nginx
    ```
25. PVC 상태 확인

    ```
    kubectl get pvc my-pvc
    ```
26. PVC에 명시된 PV가 생성되었는지 확인&#x20;

    ```
    kubectl get pv $(kubectl get pvc my-pvc -o=jsonpath='{.spec.volumeName}')
    ```
27. PVC에 명시한 볼륨 요청 크기 만큼 실제로 할당되었는지 확인

    ```
    diff <(kubectl get pvc my-pvc -o=jsonpath='{.spec}' | jq) \
    <(kubectl get pvc my-pvc -o=jsonpath='{.status}' | jq) -y
    ```
28. Deployment의 Replica 수를 10개로 수정&#x20;

    ```
    kubectl scale deploy nginx --replicas=10
    ```
29. 추가로 생성되는 Pod 상태 확인

    ```
    kubectl get pod -l app=nginx
    ```
30. Pending 상태의 Pod가 있다면 그 이유 확인

    ```
    kubectl describe pod \
    $(kubectl get pod -l app=nginx \
    -o=jsonpath='{.items[?(@.status.phase=="Pending")].metadata.name}')
    ```
31. PV에 명시된 Node Affinity 확인 - <https://kubernetes.io/docs/concepts/storage/persistent-volumes/#node-affinity>

    ```
    kubectl get pv $(kubectl get pvc my-pvc -o=jsonpath='{.spec.volumeName}') \
    -o=jsonpath='{.spec.nodeAffinity}' | jq
    ```
32. 각 Node의 가용영역 확인&#x20;

    ```
    kubectl get nodes --label-columns topology.kubernetes.io/zone
    ```
33. Pod가 배포된 Node 확인&#x20;

    ```
    kubectl get pod -l app=nginx -o wide
    ```
34. Deployment의 Replica 수를 1개로 수정&#x20;

    ```
    kubectl scale deploy nginx --replicas=1
    ```
35. Pod가 배포된 Node의 이름 확인

    ```
    kubectl get pod -l app=nginx \
    -o=jsonpath='{.items[0].spec.nodeName}{"\n"}'
    ```
36. 해당 Node에 문제가 생겼다는 상황을 가정하고 Pod를 다른 Node로 이전하고 Node 상태를 SchedulingDisabled로 변경

    ```
    kubectl drain $(kubectl get pod -l app=nginx \
    -o=jsonpath='{.items[0].spec.nodeName}') \
    --ignore-daemonsets --delete-emptydir-data
    ```
37. Pod가 생성되는지 확인

    ```
    kubectl get pod -l app=nginx
    ```
38. Pod가 Pending 상태라면 그 이유를 확인

    ```
    kubectl describe pod \
    $(kubectl get pod -l app=nginx \
    -o=jsonpath='{.items[?(@.status.phase=="Pending")].metadata.name}')
    ```
39. Node 상태 확인

    ```
    kubectl get node
    ```
40. Node에 부여된 Taint 확인

    ```
    kubectl get nodes \
    -o=custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect
    ```
41. PV 이름 확인

    ```
    kubectl get pvc my-pvc -o=jsonpath='{.spec.volumeName}{"\n"}'
    ```
42. PV에 연동된 EBS 볼륨를 확인하고 볼륨 ID를 환경변수로 지정

    ```
    {
        export VOL_ID=$(kubectl get pv $(kubectl get pvc my-pvc \
        -o=jsonpath='{.spec.volumeName}') \
        -o jsonpath='{.spec.awsElasticBlockStore.volumeID}' \
        | grep -oE "vol-[a-z0-9]+")

        echo $VOL_ID
    }
    ```
43. 위에서 확인한 EBS 볼륨 상태 확인

    ```
    aws ec2 describe-volumes --volume-ids $VOL_ID
    ```
44. Node가 정상으로 돌아왔다고 가정하고 Node에 부여된 Taint 삭제

    ```
    kubectl uncordon \
    $(kubectl get node -o=jsonpath='{.items[?(@.spec.unschedulable==true)].metadata.name}')
    ```
45. Node 상태 확인

    ```
    kubectl get node
    ```
46. Pod가 생성되는지 확인

    ```
    kubectl get pod -l app=nginx
    ```
47. PV에 연동된 EBS 볼륨 상태 확인

    ```
    aws ec2 describe-volumes --volume-ids $VOL_ID
    ```
48. 리소스 삭제

    ```
    {
        kubectl delete deploy nginx
        kubectl delete pvc my-pvc
        kubectl delete limitrange storagelimits
    }
    ```
49. PV가 삭제되었는지 확인 - *삭제 될때까지 시간이 걸릴수도 있음*

    ```
    kubectl get pv
    ```
50. PV에 연동된 EBS 볼륨이 삭제되었는지 확인

    ```
    aws ec2 describe-volumes --volume-ids $VOL_ID
    ```

### MySQL 클러스터 구성&#x20;

1. ConfigMap 생성&#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: mysql
     labels:
       app: mysql
   data:
     primary.cnf: |
       # Apply this config only on the primary.
       [mysqld]
       log-bin    
     replica.cnf: |
       # Apply this config only on replicas.
       [mysqld]
       super-read-only    
   EOF
   ```
2. Service 생성&#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: mysql
     labels:
       app: mysql
   spec:
     ports:
     - name: mysql
       port: 3306
     clusterIP: None
     selector:
       app: mysql
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: mysql-read
     labels:
       app: mysql
   spec:
     ports:
     - name: mysql
       port: 3306
     selector:
       app: mysql
   EOF
   ```
3. StatefulSet 생성

   ```
   cat <<'EOF' | kubectl apply -f -
   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: mysql
   spec:
     selector:
       matchLabels:
         app: mysql
     serviceName: mysql
     replicas: 3
     template:
       metadata:
         labels:
           app: mysql
       spec:
         initContainers:
         - name: init-mysql
           image: mysql:5.7
           command:
           - bash
           - "-c"
           - |
             set -ex
             # Generate mysql server-id from pod ordinal index.
             [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
             ordinal=${BASH_REMATCH[1]}
             echo [mysqld] > /mnt/conf.d/server-id.cnf
             # Add an offset to avoid reserved server-id=0 value.
             echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
             # Copy appropriate conf.d files from config-map to emptyDir.
             if [[ $ordinal -eq 0 ]]; then
               cp /mnt/config-map/primary.cnf /mnt/conf.d/
             else
               cp /mnt/config-map/replica.cnf /mnt/conf.d/
             fi          
           volumeMounts:
           - name: conf
             mountPath: /mnt/conf.d
           - name: config-map
             mountPath: /mnt/config-map
         - name: clone-mysql
           image: gcr.io/google-samples/xtrabackup:1.0
           command:
           - bash
           - "-c"
           - |
             set -ex
             # Skip the clone if data already exists.
             [[ -d /var/lib/mysql/mysql ]] && exit 0
             # Skip the clone on primary (ordinal index 0).
             [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
             ordinal=${BASH_REMATCH[1]}
             [[ $ordinal -eq 0 ]] && exit 0
             # Clone data from previous peer.
             ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
             # Prepare the backup.
             xtrabackup --prepare --target-dir=/var/lib/mysql          
           volumeMounts:
           - name: data
             mountPath: /var/lib/mysql
             subPath: mysql
           - name: conf
             mountPath: /etc/mysql/conf.d
         containers:
         - name: mysql
           image: mysql:5.7
           env:
           - name: MYSQL_ALLOW_EMPTY_PASSWORD
             value: "1"
           ports:
           - name: mysql
             containerPort: 3306
           volumeMounts:
           - name: data
             mountPath: /var/lib/mysql
             subPath: mysql
           - name: conf
             mountPath: /etc/mysql/conf.d
           resources:
             requests:
               cpu: 500m
               memory: 500Mi
           livenessProbe:
             exec:
               command: ["mysqladmin", "ping"]
             initialDelaySeconds: 30
             periodSeconds: 10
             timeoutSeconds: 5
           readinessProbe:
             exec:
               # Check we can execute queries over TCP (skip-networking is off).
               command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
             initialDelaySeconds: 5
             periodSeconds: 2
             timeoutSeconds: 1
         - name: xtrabackup
           image: gcr.io/google-samples/xtrabackup:1.0
           ports:
           - name: xtrabackup
             containerPort: 3307
           command:
           - bash
           - "-c"
           - |
             set -ex
             cd /var/lib/mysql

             # Determine binlog position of cloned data, if any.
             if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
               # XtraBackup already generated a partial "CHANGE MASTER TO" query
               # because we're cloning from an existing replica. (Need to remove the tailing semicolon!)
               cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
               # Ignore xtrabackup_binlog_info in this case (it's useless).
               rm -f xtrabackup_slave_info xtrabackup_binlog_info
             elif [[ -f xtrabackup_binlog_info ]]; then
               # We're cloning directly from primary. Parse binlog position.
               [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
               rm -f xtrabackup_binlog_info xtrabackup_slave_info
               echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                     MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
             fi

             # Check if we need to complete a clone by starting replication.
             if [[ -f change_master_to.sql.in ]]; then
               echo "Waiting for mysqld to be ready (accepting connections)"
               until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done

               echo "Initializing replication from clone position"
               mysql -h 127.0.0.1 \
                     -e "$(<change_master_to.sql.in), \
                             MASTER_HOST='mysql-0.mysql', \
                             MASTER_USER='root', \
                             MASTER_PASSWORD='', \
                             MASTER_CONNECT_RETRY=10; \
                           START SLAVE;" || exit 1
               # In case of container restart, attempt this at-most-once.
               mv change_master_to.sql.in change_master_to.sql.orig
             fi

             # Start a server to send backups when requested by peers.
             exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
               "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"          
           volumeMounts:
           - name: data
             mountPath: /var/lib/mysql
             subPath: mysql
           - name: conf
             mountPath: /etc/mysql/conf.d
           resources:
             requests:
               cpu: 100m
               memory: 100Mi
         volumes:
         - name: conf
           emptyDir: {}
         - name: config-map
           configMap:
             name: mysql
     volumeClaimTemplates:
     - metadata:
         name: data
       spec:
         accessModes: ["ReadWriteOnce"]
         resources:
           requests:
             storage: 10Gi
   EOF
   ```
4. Pod 생성 확인&#x20;

   ```
   watch kubectl get pod -l app=mysql
   ```
5. 생성된 PVC 확인&#x20;

   ```
   kubectl get pvc -l app=mysql
   ```
6. PVC에 명시된 PV가 생성되었는지 확인&#x20;

   ```
   kubectl get pv \
   $(kubectl get pvc -l app=mysql -o=jsonpath='{.items[*].spec.volumeName}')
   ```
7. 각 PV에 맵핑된 EBS 볼륨 확인&#x20;

   ```
   for pv in $(kubectl get pvc -l app=mysql -o=jsonpath='{.items[*].spec.volumeName}')
   do
     aws ec2 describe-volumes --filters Name=tag:kubernetes.io/created-for/pv/name,Values=$pv
   done
   ```
8. Pod 생성&#x20;

   ```
   kubectl run mysql-client --image=mysql:5.7 -- sleep 3600
   ```
9. *mysql-0* Pod를 통해서 데이터베이스 및 레코드 생성&#x20;

   ```
   kubectl exec -i mysql-client -- \
   mysql -h mysql-0.mysql <<EOF
   CREATE DATABASE test;
   CREATE TABLE test.messages (message VARCHAR(250));
   INSERT INTO test.messages VALUES ('hello');
   EOF
   ```
10. 읽기 엔드포인트 *mysql-read* 통해서 위에서 생성한 데이터를 확인&#x20;

    ```
    kubectl exec -it mysql-client -- \
    mysql -h mysql-read -e "SELECT (SELECT message FROM test.messages) as message, (SELECT @@server_id) server_id;"
    ```
11. 읽기 엔드포인트 *mysql-read* 가 읽기에 대한 쿼리를 분산하는지 확인&#x20;

    ```
    kubectl exec -it mysql-client -- \
    bash -c "while sleep 1; do mysql -h mysql-read -e 'SELECT (SELECT message FROM test.messages) as message, (SELECT @@server_id) server_id;'; done"
    ```
12. Ctrl+C를 입력해서 위에서 실행한 프로세스 종료
13. *mysql-read* Endpoint에 타켓으로 포함되어 있는 Pod 목록 확인&#x20;

    ```
    kubectl get ep mysql-read \
    -o jsonpath='{.subsets[*].addresses[*].targetRef.name}{"\n"}'
    ```
14. *mysql-2* Pod의 ReadinessProbe가 실패하도록 수정&#x20;

    ```
    {
        kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off
        kubectl exec mysql-2 -c mysql -- cp /usr/bin/bash /usr/bin/mysql
    }
    ```
15. *mysql-2* Pod의 상태 확인&#x20;

    ```
    kubectl get pod mysql-2
    ```
16. *mysql-2* Pod의 상세 상태 확인&#x20;

    ```
    kubectl describe pod mysql-2
    ```
17. 읽기 엔드포인트 *mysql-read* 로 쿼리를 실행해서 *mysql-2* Pod로 트래픽이 전달되는지 확인&#x20;

    ```
    kubectl exec -it mysql-client -- \
    bash -c "while sleep 1; do mysql -h mysql-read -e 'SELECT (SELECT message FROM test.messages) as message, (SELECT @@server_id) server_id;'; done"
    ```
18. *mysql-read* Endpoint에 타켓으로 *mysql-2* Pod가 포함되어 있는지 확인&#x20;

    ```
    kubectl get ep mysql-read \
    -o jsonpath='{.subsets[*].addresses[*].targetRef.name}{"\n"}'
    ```
19. *mysql-2* Pod의 ReadinessProbe가 성공하도록 수정&#x20;

    ```
    kubectl exec mysql-2 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql
    ```
20. *mysql-2* Pod의 상태 확인&#x20;

    ```
    kubectl get pod mysql-2
    ```
21. *mysql-read* Endpoint에 타켓으로 *mysql-2* Pod가 포함되어 있는지 확인&#x20;

    ```
    kubectl get ep mysql-read \
    -o jsonpath='{.subsets[*].addresses[*].targetRef.name}{"\n"}'
    ```
22. StatefulSet의 Replica 갯수를 0개로 조정&#x20;

    ```
    kubectl scale sts mysql --replicas=0
    ```
23. Pod 상태 확인&#x20;

    ```
    watch kubectl get pod -l app=mysql
    ```
24. PVC가 삭제되었는지 확인&#x20;

    ```
    kubectl get pvc -l app=mysql 
    ```
25. StatefulSet의 Replica 갯수를 3개로 조정&#x20;

    ```
    kubectl scale sts mysql --replicas=3
    ```
26. Pod 상태 확인&#x20;

    ```
    watch kubectl get pod -l app=mysql
    ```
27. 새로운 PVC가 생성되었는지 확인&#x20;

    ```
    kubectl get pvc -l app=mysql
    ```
28. 기존에 생성한 데이터베이스 및 레코드가 존재하는지 확인&#x20;

    ```
    kubectl exec -it mysql-client -- \
    bash -c "while sleep 1; do mysql -h mysql-read -e 'SELECT (SELECT message FROM test.messages) as message, (SELECT @@server_id) server_id;'; done"
    ```
29. Ctrl+C를 입력해서 위에서 실행한 프로세스 종료
30. StatefulSet 삭제&#x20;

    ```
    kubectl delete sts mysql
    ```
31. PVC가 삭제되었는지 확인&#x20;

    ```
    kubectl get pvc -l app=mysql
    ```
32. StatefulSet 재생성&#x20;

    ```
    cat <<'EOF' | kubectl apply -f -
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: mysql
    spec:
      selector:
        matchLabels:
          app: mysql
      serviceName: mysql
      replicas: 3
      template:
        metadata:
          labels:
            app: mysql
        spec:
          initContainers:
          - name: init-mysql
            image: mysql:5.7
            command:
            - bash
            - "-c"
            - |
              set -ex
              # Generate mysql server-id from pod ordinal index.
              [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
              ordinal=${BASH_REMATCH[1]}
              echo [mysqld] > /mnt/conf.d/server-id.cnf
              # Add an offset to avoid reserved server-id=0 value.
              echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
              # Copy appropriate conf.d files from config-map to emptyDir.
              if [[ $ordinal -eq 0 ]]; then
                cp /mnt/config-map/primary.cnf /mnt/conf.d/
              else
                cp /mnt/config-map/replica.cnf /mnt/conf.d/
              fi          
            volumeMounts:
            - name: conf
              mountPath: /mnt/conf.d
            - name: config-map
              mountPath: /mnt/config-map
          - name: clone-mysql
            image: gcr.io/google-samples/xtrabackup:1.0
            command:
            - bash
            - "-c"
            - |
              set -ex
              # Skip the clone if data already exists.
              [[ -d /var/lib/mysql/mysql ]] && exit 0
              # Skip the clone on primary (ordinal index 0).
              [[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
              ordinal=${BASH_REMATCH[1]}
              [[ $ordinal -eq 0 ]] && exit 0
              # Clone data from previous peer.
              ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
              # Prepare the backup.
              xtrabackup --prepare --target-dir=/var/lib/mysql          
            volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
              subPath: mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
          containers:
          - name: mysql
            image: mysql:5.7
            env:
            - name: MYSQL_ALLOW_EMPTY_PASSWORD
              value: "1"
            ports:
            - name: mysql
              containerPort: 3306
            volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
              subPath: mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
            resources:
              requests:
                cpu: 500m
                memory: 500Mi
            livenessProbe:
              exec:
                command: ["mysqladmin", "ping"]
              initialDelaySeconds: 30
              periodSeconds: 10
              timeoutSeconds: 5
            readinessProbe:
              exec:
                # Check we can execute queries over TCP (skip-networking is off).
                command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
              initialDelaySeconds: 5
              periodSeconds: 2
              timeoutSeconds: 1
          - name: xtrabackup
            image: gcr.io/google-samples/xtrabackup:1.0
            ports:
            - name: xtrabackup
              containerPort: 3307
            command:
            - bash
            - "-c"
            - |
              set -ex
              cd /var/lib/mysql

              # Determine binlog position of cloned data, if any.
              if [[ -f xtrabackup_slave_info && "x$(<xtrabackup_slave_info)" != "x" ]]; then
                # XtraBackup already generated a partial "CHANGE MASTER TO" query
                # because we're cloning from an existing replica. (Need to remove the tailing semicolon!)
                cat xtrabackup_slave_info | sed -E 's/;$//g' > change_master_to.sql.in
                # Ignore xtrabackup_binlog_info in this case (it's useless).
                rm -f xtrabackup_slave_info xtrabackup_binlog_info
              elif [[ -f xtrabackup_binlog_info ]]; then
                # We're cloning directly from primary. Parse binlog position.
                [[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
                rm -f xtrabackup_binlog_info xtrabackup_slave_info
                echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
                      MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
              fi

              # Check if we need to complete a clone by starting replication.
              if [[ -f change_master_to.sql.in ]]; then
                echo "Waiting for mysqld to be ready (accepting connections)"
                until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done

                echo "Initializing replication from clone position"
                mysql -h 127.0.0.1 \
                      -e "$(<change_master_to.sql.in), \
                              MASTER_HOST='mysql-0.mysql', \
                              MASTER_USER='root', \
                              MASTER_PASSWORD='', \
                              MASTER_CONNECT_RETRY=10; \
                            START SLAVE;" || exit 1
                # In case of container restart, attempt this at-most-once.
                mv change_master_to.sql.in change_master_to.sql.orig
              fi

              # Start a server to send backups when requested by peers.
              exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
                "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"          
            volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
              subPath: mysql
            - name: conf
              mountPath: /etc/mysql/conf.d
            resources:
              requests:
                cpu: 100m
                memory: 100Mi
          volumes:
          - name: conf
            emptyDir: {}
          - name: config-map
            configMap:
              name: mysql
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi
    EOF
    ```
33. Pod 생성 확인&#x20;

    ```
    watch kubectl get pod -l app=mysql
    ```
34. 새로운 PVC가 생성되었는지 확인&#x20;

    ```
    kubectl get pvc -l app=mysql
    ```
35. 기존에 생성한 데이터베이스 및 레코드가 존재하는지 확인&#x20;

    ```
    kubectl exec -it mysql-client -- \
    bash -c "while sleep 1; do mysql -h mysql-read -e 'SELECT (SELECT message FROM test.messages) as message, (SELECT @@server_id) server_id;'; done"
    ```
36. Ctrl+C를 입력해서 위에서 실행한 프로세스 종료
37. 리소스 삭제 &#x20;

    ```
    {
        kubectl delete sts mysql
        kubectl delete pvc -l app=mysql
        kubectl delete svc mysql mysql-read
        kubectl delete cm mysql
        kubectl delete pod mysql-client
    }
    ```