> For the complete documentation index, see [llms.txt](https://kubernetes.youngwjung.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kubernetes.youngwjung.com/extras/ceph/undefined.md).

# 실습

### 사전 작업

1. Ingress-NGINX 컨트롤러 설치
2. ExternalDNS 설치

### Ceph 클러스터 구성

1. Node에 부여된 Label을 통해서 EKS 클러스터 이름 확인하고 환경변수로 저장

   ```
   {
       export CLUSTER_NAME=$(kubectl get node \
       -o=jsonpath='{.items[0].metadata.labels.alpha\.eksctl\.io\/cluster-name}')
       echo $CLUSTER_NAME
   }
   ```
2. Ceph 클러스터에 사용할 노드그룹에 생성 

   ```
   {
       cat <<EOF > ceph-node.yaml
   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig

   metadata:
     name: $CLUSTER_NAME
     region: ap-northeast-2

   managedNodeGroups:
   - name: ceph-node
     instanceType: c5.2xlarge
     desiredCapacity: 3
     volumeSize: 20
     labels:
       role: ceph
     additionalVolumes:
     - volumeName: /dev/sdf
       volumeSize: 20
       volumeType: gp3
   EOF
       eksctl create nodegroup --config-file=ceph-node.yaml
   }
   ```
3. 생성된 노드 확인&#x20;

   ```
   kubectl get node -l role=ceph
   ```
4. Namespace 생성&#x20;

   ```
   kubectl create ns rook-ceph
   ```
5. Helm 차트 리포지토리 추가&#x20;

   ```
   {
       helm repo add rook-release https://charts.rook.io/release
       helm repo update
   }
   ```
6. Ceph Operator 차트에 지정할 변수를 명시한 파일 생성 - <https://rook.io/docs/rook/v1.12/Helm-Charts/operator-chart/#configuration>

   ```
   cat <<EOF > rook-ceph.yaml
   nodeSelector:
     role: ceph
   csi:
     provisionerNodeAffinity: role=ceph
   admissionController:
     nodeAffinity: role=ceph
   priorityClassName: system-cluster-critical
   EOF
   ```
7. Ceph Operator 설치&#x20;

   ```
   helm install rook-ceph rook-release/rook-ceph -n rook-ceph -f rook-ceph.yaml
   ```
8. Ceph Operator 상태 확인&#x20;

   ```
   kubectl -n rook-ceph get pod -l app=rook-ceph-operator
   ```
9. Ceph Operator 로그 확인&#x20;

   ```
   kubectl -n rook-ceph logs deploy/rook-ceph-operator
   ```
10. Ceph Cluster 차트에 지정할 변수를 명시한 파일 생성 - <https://rook.io/docs/rook/v1.12/Helm-Charts/ceph-cluster-chart/#configuration>

    ```
    cat <<EOF > rook-ceph-cluster.yaml
    cephClusterSpec:
      cephVersion:
        image: quay.io/ceph/ceph:v17.2.5
      storage:
        useAllNodes: false
        useAllDevices: false
        nodes:
        - name: NODE_NAME
          devices:
          - name: DEVICE_NAME
        - name: NODE_NAME
          devices:
          - name: DEVICE_NAME
        - name: NODE_NAME
          devices:
          - name: DEVICE_NAME
      placement:
        all:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - ceph
      crashCollector:
        disable: true
      dashboard:
        enabled: true
        ssl: false
    cephFileSystems:
    - name: ceph-filesystem
      spec:
        metadataPool:
          replicated:
            size: 3
        dataPools:
        - failureDomain: host
          replicated:
            size: 3
          name: data0
        metadataServer:
          activeCount: 1
          activeStandby: true
          resources:
            limits:
              cpu: "2000m"
              memory: "4Gi"
            requests:
              cpu: "1000m"
              memory: "4Gi"
          priorityClassName: system-cluster-critical
          placement:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - ceph
      storageClass:
        enabled: true
        isDefault: false
        name: ceph-filesystem
        pool: data0
        reclaimPolicy: Delete
        allowVolumeExpansion: true
        volumeBindingMode: "Immediate"
        mountOptions: []
        parameters:
          csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
          csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
          csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
          csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
          csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
          csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"
          csi.storage.k8s.io/fstype: ext4
    cephObjectStores:
    - name: ceph-objectstore
      spec:
        metadataPool:
          failureDomain: host
          replicated:
            size: 3
        dataPool:
          failureDomain: host
          erasureCoded:
            dataChunks: 2
            codingChunks: 1
        preservePoolsOnDelete: true
        gateway:
          port: 80
          resources:
            limits:
              cpu: "2000m"
              memory: "2Gi"
            requests:
              cpu: "1000m"
              memory: "1Gi"
          instances: 1
          priorityClassName: system-cluster-critical
          placement:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - ceph
      storageClass:
        enabled: true
        name: ceph-bucket
        reclaimPolicy: Delete
        volumeBindingMode: "Immediate"
        parameters:
          region: ap-northeast-2
    ingress:
      dashboard:
        ingressClassName: nginx
        host:
          name: HOST_NAME
    toolbox:
      enabled: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: role
                  operator: In
                  values:
                  - ceph
    monitoring:
      enabled: false
      createPrometheusRules: false
    EOF
    ```
11. 위에서 생성한 파일을 열고 NODE\_NAME, DEV\_NAME, HOST\_NAME을 변경&#x20;
12. Ceph Cluster 설치&#x20;

    ```
    helm install rook-ceph-cluster rook-release/rook-ceph-cluster -n rook-ceph -f rook-ceph-cluster.yaml
    ```
13. 생성된 CRD 확인&#x20;

    ```
    kubectl get crd
    ```
14. CephCluster 객체 확인&#x20;

    ```
    kubectl get cephclusters.ceph.rook.io rook-ceph -n rook-ceph
    ```
15. Ceph Operator 로그 확인&#x20;

    ```
    kubectl -n rook-ceph logs deploy/rook-ceph-operator -f
    ```
16. 생성된 Pod 목록 확인&#x20;

    ```
    kubectl get pod -n rook-ceph
    ```
17. Ceph Operator 로그 확인&#x20;

    ```
    kubectl -n rook-ceph logs deploy/rook-ceph-operator -f
    ```
18. Ceph Cluster 대시보드에 연동된 Ingress 객체 확인&#x20;

    ```
    kubectl get ing -n rook-ceph
    ```
19. 대시보드 로그인에 필요한 비밀번호 확인&#x20;

    ```
    kubectl -n rook-ceph get secret rook-ceph-dashboard-password \
    -o jsonpath={.data.password} | base64 -d && echo
    ```
20. 웹브라우저를 열고 대시보드로 연결 후 로그인&#x20;
21. 클러스터 상태가 정상인지 확인&#x20;
22. StorageClass가 생성되었는지 확인&#x20;

    ```
    kubectl get sc
    ```

### 볼륨 프로비저닝

1. 생성된 PVC가 있는지 확인&#x20;

   ```
   kubectl get pvc
   ```
2. PVC 생성 &#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: nginx-data
   spec:
     storageClassName: ceph-block
     accessModes:
     - ReadWriteOnce
     resources:
        requests:
          storage: 1Gi
   EOF
   ```
3. PVC가 생성되었는지 확인&#x20;

   ```
   kubectl get pvc nginx-data
   ```
4. PVC에 발생한 이벤트 확인&#x20;

   ```
   kubectl describe pvc nginx-data
   ```
5. PV가 생성되었는지 확인&#x20;

   ```
   kubectl get pv
   ```
6. 생성된 PV의 상세 내용 확인&#x20;

   ```
   kubectl get pv -o yaml
   ```
7. Ceph Cluster 대시보드로 이동해서 새로운 Block 스토리지가 생성되었는지 확인&#x20;
8. Deployment 생성&#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       run: nginx
     name: nginx
   spec:
     selector:
       matchLabels:
         run: nginx
     template:
       metadata:
         labels:
           run: nginx
       spec:
         containers:
         - image: nginx
           imagePullPolicy: Always
           name: nginx
           volumeMounts:
           - name: data
             mountPath: /data
           ports:
           - containerPort: 80
             protocol: TCP
         volumes:
         - name: data
           persistentVolumeClaim:
             claimName: nginx-data
   EOF
   ```
9. 생성된 Pod 확인&#x20;

   ```
   kubectl get pod -l run=nginx
   ```
10. PV를 마운트한 경로에 파일 생성&#x20;

    ```
    kubectl exec -it deploy/nginx -- bash -c "echo hello > /data/world.txt"
    ```
11. 파일이 정상적으로 생성되었는지 확인&#x20;

    ```
    kubectl exec -it deploy/nginx -- cat /data/world.txt
    ```
12. PVC 수정&#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: nginx-data
    spec:
      storageClassName: ceph-block
      accessModes:
      - ReadWriteOnce
      resources:
         requests:
           storage: 2Gi
    EOF
    ```
13. PVC가 수정되었는지 확인&#x20;

    ```
    kubectl get pvc nginx-data
    ```
14. PVC에 발생한 이벤트 확인&#x20;

    ```
    kubectl describe pvc nginx-data
    ```
15. PV가 수정되었는지 확인&#x20;

    ```
    kubectl get pv
    ```
16. Ceph Cluster 대시보드로 이동해서 생성된 Block 스토리지에 변경 사항이 있는지 확인&#x20;
17. Deployment의 복제본 갯수를 2개로 수정&#x20;

    ```
    kubectl scale deploy nginx --replicas=2
    ```
18. Pod가 생성되었는지 확인&#x20;

    ```
    kubectl get pod -l run=nginx
    ```
19. 새로운 Pod가 실행되지 않는 이유 확인&#x20;

    ```
    kubectl describe pod \
    $(kubectl get pod -l run=nginx --sort-by='.metadata.creationTimestamp' -o=jsonpath='{.items[-1].metadata.name}')
    ```
20. PVC 삭제&#x20;

    ```
    kubectl delete pvc nginx-data
    ```
21. 위의 명령어가 종료되지 않을 경우에는 Ctrl+C를 입력&#x20;
22. PVC 상태 확인&#x20;

    ```
    kubectl get pvc nginx-data
    ```
23. PVC가 삭제되지 않는 이유 확인&#x20;

    ```
    kubectl get pvc nginx-data -o yaml
    ```
24. PV 상태 확인&#x20;

    ```
    kubectl get pv
    ```
25. Deployment 삭제&#x20;

    ```
    kubectl delete deploy nginx
    ```
26. PVC가 삭제되었는지 확인&#x20;

    <pre><code><strong>kubectl get pvc nginx-data
    </strong></code></pre>
27. PV가 삭제되었는지 확인&#x20;

    ```
    kubectl get pv
    ```
28. PVC 생성 &#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: nginx-data
    spec:
      storageClassName: ceph-filesystem
      accessModes:
      - ReadWriteMany
      resources:
         requests:
           storage: 1Gi
    EOF
    ```
29. PVC가 생성되었는지 확인&#x20;

    ```
    kubectl get pvc nginx-data
    ```
30. PVC에 발생한 이벤트 확인&#x20;

    ```
    kubectl describe pvc nginx-data
    ```
31. PV가 생성되었는지 확인&#x20;

    ```
    kubectl get pv
    ```
32. 생성된 PV의 상세 내용 확인&#x20;

    ```
    kubectl get pv -o yaml
    ```
33. Ceph Cluster 대시보드로 이동해서 새로운 Filesystem이 생성되었는지 확인&#x20;
34. Deployment 생성&#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        run: nginx
      name: nginx
    spec:
      selector:
        matchLabels:
          run: nginx
      template:
        metadata:
          labels:
            run: nginx
        spec:
          containers:
          - image: nginx
            imagePullPolicy: Always
            name: nginx
            volumeMounts:
            - name: data
              mountPath: /data
            ports:
            - containerPort: 80
              protocol: TCP
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: nginx-data
    EOF
    ```
35. 생성된 Pod 확인&#x20;

    ```
    kubectl get pod -l run=nginx
    ```
36. PV를 마운트한 경로에 파일 생성&#x20;

    ```
    kubectl exec -it deploy/nginx -- bash -c "echo hello > /data/world.txt"
    ```
37. 파일이 정상적으로 생성되었는지 확인&#x20;

    ```
    kubectl exec -it deploy/nginx -- cat /data/world.txt
    ```
38. Deployment의 복제본 갯수를 2개로 수정&#x20;

    ```
    kubectl scale deploy nginx --replicas=2
    ```
39. Pod가 생성되었는지 확인&#x20;

    ```
    kubectl get pod -l run=nginx
    ```
40. 새로 생성된 Pod에서 다른 Pod에서 생성한 파일이 보이는지 확인&#x20;

    ```
    kubectl exec -it $(kubectl get pod -l run=nginx --sort-by='.metadata.creationTimestamp' -o=jsonpath='{.items[-1].metadata.name}') \
    -- cat /data/world.txt
    ```
41. 리소스 삭제&#x20;

    ```
    {
        kubectl delete deploy nginx
        kubectl delete pvc nginx-data
    }
    ```
42. PV 목록 확인&#x20;

    ```
    kubectl get pv
    ```

### Cleanup

1. Ceph Cluster 삭제&#x20;

   ```
   helm uninstall rook-ceph-cluster -n rook-ceph
   ```
2. Ceph Operator 로그 확인&#x20;

   ```
   kubectl -n rook-ceph logs deploy/rook-ceph-operator -f
   ```
3. Pod 목록 확인&#x20;

   ```
   kubectl get pod -n rook-ceph
   ```
4. Ceph Operator 삭제&#x20;

   ```
   helm uninstall rook-ceph -n rook-ceph
   ```
5. Pod 목록 확인&#x20;

   ```
   kubectl get pod -n rook-ceph
   ```
6. Namespace 삭제&#x20;

   ```
   kubectl delete ns rook-ceph
   ```
7. 노드그룹 삭제&#x20;

   ```
   eksctl delete nodegroup --config-file=ceph-node.yaml --approve
   ```
8. 노드 목록 확인&#x20;

   ```
   kubectl get node
   ```