# 실습

### 사전 작업

1. Ingress-NGINX 컨트롤러 설치
2. ExternalDNS 설치

### Ceph 클러스터 구성

1. Node에 부여된 Label을 통해서 EKS 클러스터 이름 확인하고 환경변수로 저장

   ```
   {
       export CLUSTER_NAME=$(kubectl get node \
       -o=jsonpath='{.items[0].metadata.labels.alpha\.eksctl\.io\/cluster-name}')
       echo $CLUSTER_NAME
   }
   ```
2. Ceph 클러스터에 사용할 노드그룹에 생성 

   ```
   {
       cat <<EOF > ceph-node.yaml
   apiVersion: eksctl.io/v1alpha5
   kind: ClusterConfig

   metadata:
     name: $CLUSTER_NAME
     region: ap-northeast-2

   managedNodeGroups:
   - name: ceph-node
     instanceType: c5.2xlarge
     desiredCapacity: 3
     volumeSize: 20
     labels:
       role: ceph
     additionalVolumes:
     - volumeName: /dev/sdf
       volumeSize: 20
       volumeType: gp3
   EOF
       eksctl create nodegroup --config-file=ceph-node.yaml
   }
   ```
3. 생성된 노드 확인&#x20;

   ```
   kubectl get node -l role=ceph
   ```
4. Namespace 생성&#x20;

   ```
   kubectl create ns rook-ceph
   ```
5. Helm 차트 리포지토리 추가&#x20;

   ```
   {
       helm repo add rook-release https://charts.rook.io/release
       helm repo update
   }
   ```
6. Ceph Operator 차트에 지정할 변수를 명시한 파일 생성 - <https://rook.io/docs/rook/v1.12/Helm-Charts/operator-chart/#configuration>

   ```
   cat <<EOF > rook-ceph.yaml
   nodeSelector:
     role: ceph
   csi:
     provisionerNodeAffinity: role=ceph
   admissionController:
     nodeAffinity: role=ceph
   priorityClassName: system-cluster-critical
   EOF
   ```
7. Ceph Operator 설치&#x20;

   ```
   helm install rook-ceph rook-release/rook-ceph -n rook-ceph -f rook-ceph.yaml
   ```
8. Ceph Operator 상태 확인&#x20;

   ```
   kubectl -n rook-ceph get pod -l app=rook-ceph-operator
   ```
9. Ceph Operator 로그 확인&#x20;

   ```
   kubectl -n rook-ceph logs deploy/rook-ceph-operator
   ```
10. Ceph Cluster 차트에 지정할 변수를 명시한 파일 생성 - <https://rook.io/docs/rook/v1.12/Helm-Charts/ceph-cluster-chart/#configuration>

    ```
    cat <<EOF > rook-ceph-cluster.yaml
    cephClusterSpec:
      cephVersion:
        image: quay.io/ceph/ceph:v17.2.5
      storage:
        useAllNodes: false
        useAllDevices: false
        nodes:
        - name: NODE_NAME
          devices:
          - name: DEVICE_NAME
        - name: NODE_NAME
          devices:
          - name: DEVICE_NAME
        - name: NODE_NAME
          devices:
          - name: DEVICE_NAME
      placement:
        all:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - ceph
      crashCollector:
        disable: true
      dashboard:
        enabled: true
        ssl: false
    cephFileSystems:
    - name: ceph-filesystem
      spec:
        metadataPool:
          replicated:
            size: 3
        dataPools:
        - failureDomain: host
          replicated:
            size: 3
          name: data0
        metadataServer:
          activeCount: 1
          activeStandby: true
          resources:
            limits:
              cpu: "2000m"
              memory: "4Gi"
            requests:
              cpu: "1000m"
              memory: "4Gi"
          priorityClassName: system-cluster-critical
          placement:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - ceph
      storageClass:
        enabled: true
        isDefault: false
        name: ceph-filesystem
        pool: data0
        reclaimPolicy: Delete
        allowVolumeExpansion: true
        volumeBindingMode: "Immediate"
        mountOptions: []
        parameters:
          csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
          csi.storage.k8s.io/provisioner-secret-namespace: "{{ .Release.Namespace }}"
          csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
          csi.storage.k8s.io/controller-expand-secret-namespace: "{{ .Release.Namespace }}"
          csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
          csi.storage.k8s.io/node-stage-secret-namespace: "{{ .Release.Namespace }}"
          csi.storage.k8s.io/fstype: ext4
    cephObjectStores:
    - name: ceph-objectstore
      spec:
        metadataPool:
          failureDomain: host
          replicated:
            size: 3
        dataPool:
          failureDomain: host
          erasureCoded:
            dataChunks: 2
            codingChunks: 1
        preservePoolsOnDelete: true
        gateway:
          port: 80
          resources:
            limits:
              cpu: "2000m"
              memory: "2Gi"
            requests:
              cpu: "1000m"
              memory: "1Gi"
          instances: 1
          priorityClassName: system-cluster-critical
          placement:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: role
                    operator: In
                    values:
                    - ceph
      storageClass:
        enabled: true
        name: ceph-bucket
        reclaimPolicy: Delete
        volumeBindingMode: "Immediate"
        parameters:
          region: ap-northeast-2
    ingress:
      dashboard:
        ingressClassName: nginx
        host:
          name: HOST_NAME
    toolbox:
      enabled: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: role
                  operator: In
                  values:
                  - ceph
    monitoring:
      enabled: false
      createPrometheusRules: false
    EOF
    ```
11. 위에서 생성한 파일을 열고 NODE\_NAME, DEV\_NAME, HOST\_NAME을 변경&#x20;
12. Ceph Cluster 설치&#x20;

    ```
    helm install rook-ceph-cluster rook-release/rook-ceph-cluster -n rook-ceph -f rook-ceph-cluster.yaml
    ```
13. 생성된 CRD 확인&#x20;

    ```
    kubectl get crd
    ```
14. CephCluster 객체 확인&#x20;

    ```
    kubectl get cephclusters.ceph.rook.io rook-ceph -n rook-ceph
    ```
15. Ceph Operator 로그 확인&#x20;

    ```
    kubectl -n rook-ceph logs deploy/rook-ceph-operator -f
    ```
16. 생성된 Pod 목록 확인&#x20;

    ```
    kubectl get pod -n rook-ceph
    ```
17. Ceph Operator 로그 확인&#x20;

    ```
    kubectl -n rook-ceph logs deploy/rook-ceph-operator -f
    ```
18. Ceph Cluster 대시보드에 연동된 Ingress 객체 확인&#x20;

    ```
    kubectl get ing -n rook-ceph
    ```
19. 대시보드 로그인에 필요한 비밀번호 확인&#x20;

    ```
    kubectl -n rook-ceph get secret rook-ceph-dashboard-password \
    -o jsonpath={.data.password} | base64 -d && echo
    ```
20. 웹브라우저를 열고 대시보드로 연결 후 로그인&#x20;
21. 클러스터 상태가 정상인지 확인&#x20;
22. StorageClass가 생성되었는지 확인&#x20;

    ```
    kubectl get sc
    ```

### 볼륨 프로비저닝

1. 생성된 PVC가 있는지 확인&#x20;

   ```
   kubectl get pvc
   ```
2. PVC 생성 &#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: nginx-data
   spec:
     storageClassName: ceph-block
     accessModes:
     - ReadWriteOnce
     resources:
        requests:
          storage: 1Gi
   EOF
   ```
3. PVC가 생성되었는지 확인&#x20;

   ```
   kubectl get pvc nginx-data
   ```
4. PVC에 발생한 이벤트 확인&#x20;

   ```
   kubectl describe pvc nginx-data
   ```
5. PV가 생성되었는지 확인&#x20;

   ```
   kubectl get pv
   ```
6. 생성된 PV의 상세 내용 확인&#x20;

   ```
   kubectl get pv -o yaml
   ```
7. Ceph Cluster 대시보드로 이동해서 새로운 Block 스토리지가 생성되었는지 확인&#x20;
8. Deployment 생성&#x20;

   ```
   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       run: nginx
     name: nginx
   spec:
     selector:
       matchLabels:
         run: nginx
     template:
       metadata:
         labels:
           run: nginx
       spec:
         containers:
         - image: nginx
           imagePullPolicy: Always
           name: nginx
           volumeMounts:
           - name: data
             mountPath: /data
           ports:
           - containerPort: 80
             protocol: TCP
         volumes:
         - name: data
           persistentVolumeClaim:
             claimName: nginx-data
   EOF
   ```
9. 생성된 Pod 확인&#x20;

   ```
   kubectl get pod -l run=nginx
   ```
10. PV를 마운트한 경로에 파일 생성&#x20;

    ```
    kubectl exec -it deploy/nginx -- bash -c "echo hello > /data/world.txt"
    ```
11. 파일이 정상적으로 생성되었는지 확인&#x20;

    ```
    kubectl exec -it deploy/nginx -- cat /data/world.txt
    ```
12. PVC 수정&#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: nginx-data
    spec:
      storageClassName: ceph-block
      accessModes:
      - ReadWriteOnce
      resources:
         requests:
           storage: 2Gi
    EOF
    ```
13. PVC가 수정되었는지 확인&#x20;

    ```
    kubectl get pvc nginx-data
    ```
14. PVC에 발생한 이벤트 확인&#x20;

    ```
    kubectl describe pvc nginx-data
    ```
15. PV가 수정되었는지 확인&#x20;

    ```
    kubectl get pv
    ```
16. Ceph Cluster 대시보드로 이동해서 생성된 Block 스토리지에 변경 사항이 있는지 확인&#x20;
17. Deployment의 복제본 갯수를 2개로 수정&#x20;

    ```
    kubectl scale deploy nginx --replicas=2
    ```
18. Pod가 생성되었는지 확인&#x20;

    ```
    kubectl get pod -l run=nginx
    ```
19. 새로운 Pod가 실행되지 않는 이유 확인&#x20;

    ```
    kubectl describe pod \
    $(kubectl get pod -l run=nginx --sort-by='.metadata.creationTimestamp' -o=jsonpath='{.items[-1].metadata.name}')
    ```
20. PVC 삭제&#x20;

    ```
    kubectl delete pvc nginx-data
    ```
21. 위의 명령어가 종료되지 않을 경우에는 Ctrl+C를 입력&#x20;
22. PVC 상태 확인&#x20;

    ```
    kubectl get pvc nginx-data
    ```
23. PVC가 삭제되지 않는 이유 확인&#x20;

    ```
    kubectl get pvc nginx-data -o yaml
    ```
24. PV 상태 확인&#x20;

    ```
    kubectl get pv
    ```
25. Deployment 삭제&#x20;

    ```
    kubectl delete deploy nginx
    ```
26. PVC가 삭제되었는지 확인&#x20;

    <pre><code><strong>kubectl get pvc nginx-data
    </strong></code></pre>
27. PV가 삭제되었는지 확인&#x20;

    ```
    kubectl get pv
    ```
28. PVC 생성 &#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: nginx-data
    spec:
      storageClassName: ceph-filesystem
      accessModes:
      - ReadWriteMany
      resources:
         requests:
           storage: 1Gi
    EOF
    ```
29. PVC가 생성되었는지 확인&#x20;

    ```
    kubectl get pvc nginx-data
    ```
30. PVC에 발생한 이벤트 확인&#x20;

    ```
    kubectl describe pvc nginx-data
    ```
31. PV가 생성되었는지 확인&#x20;

    ```
    kubectl get pv
    ```
32. 생성된 PV의 상세 내용 확인&#x20;

    ```
    kubectl get pv -o yaml
    ```
33. Ceph Cluster 대시보드로 이동해서 새로운 Filesystem이 생성되었는지 확인&#x20;
34. Deployment 생성&#x20;

    ```
    cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        run: nginx
      name: nginx
    spec:
      selector:
        matchLabels:
          run: nginx
      template:
        metadata:
          labels:
            run: nginx
        spec:
          containers:
          - image: nginx
            imagePullPolicy: Always
            name: nginx
            volumeMounts:
            - name: data
              mountPath: /data
            ports:
            - containerPort: 80
              protocol: TCP
          volumes:
          - name: data
            persistentVolumeClaim:
              claimName: nginx-data
    EOF
    ```
35. 생성된 Pod 확인&#x20;

    ```
    kubectl get pod -l run=nginx
    ```
36. PV를 마운트한 경로에 파일 생성&#x20;

    ```
    kubectl exec -it deploy/nginx -- bash -c "echo hello > /data/world.txt"
    ```
37. 파일이 정상적으로 생성되었는지 확인&#x20;

    ```
    kubectl exec -it deploy/nginx -- cat /data/world.txt
    ```
38. Deployment의 복제본 갯수를 2개로 수정&#x20;

    ```
    kubectl scale deploy nginx --replicas=2
    ```
39. Pod가 생성되었는지 확인&#x20;

    ```
    kubectl get pod -l run=nginx
    ```
40. 새로 생성된 Pod에서 다른 Pod에서 생성한 파일이 보이는지 확인&#x20;

    ```
    kubectl exec -it $(kubectl get pod -l run=nginx --sort-by='.metadata.creationTimestamp' -o=jsonpath='{.items[-1].metadata.name}') \
    -- cat /data/world.txt
    ```
41. 리소스 삭제&#x20;

    ```
    {
        kubectl delete deploy nginx
        kubectl delete pvc nginx-data
    }
    ```
42. PV 목록 확인&#x20;

    ```
    kubectl get pv
    ```

### Cleanup

1. Ceph Cluster 삭제&#x20;

   ```
   helm uninstall rook-ceph-cluster -n rook-ceph
   ```
2. Ceph Operator 로그 확인&#x20;

   ```
   kubectl -n rook-ceph logs deploy/rook-ceph-operator -f
   ```
3. Pod 목록 확인&#x20;

   ```
   kubectl get pod -n rook-ceph
   ```
4. Ceph Operator 삭제&#x20;

   ```
   helm uninstall rook-ceph -n rook-ceph
   ```
5. Pod 목록 확인&#x20;

   ```
   kubectl get pod -n rook-ceph
   ```
6. Namespace 삭제&#x20;

   ```
   kubectl delete ns rook-ceph
   ```
7. 노드그룹 삭제&#x20;

   ```
   eksctl delete nodegroup --config-file=ceph-node.yaml --approve
   ```
8. 노드 목록 확인&#x20;

   ```
   kubectl get node
   ```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://kubernetes.youngwjung.com/extras/ceph/undefined.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.