실습

DaemonSet

1. 현재 생성되어 있는 DaemonSet 확인

   kubectl get ds -A

2. DaemonSet를 통해서 생성된 Pod들이 배포된 Node 확인

   kubectl get pod -l 'k8s-app in (kube-proxy, aws-node)' -A -o wide

3. Node 1개 추가

   eksctl scale nodegroup --cluster=mycluster --nodes=3 nodegroup

4. Node가 추가 되었는지 확인 - 생성 될때까지 시간이 걸릴수도 있음

   kubectl get node

5. DaemonSet를 통해서 생성되는 Pod들이 새롭게 생성된 Node에 배포되었는지 확인

   kubectl get pod -l 'k8s-app in (kube-proxy, aws-node)' -A -o wide

6. Node 갯수를 원래대로 조정

   eksctl scale nodegroup --cluster=mycluster --nodes=2 nodegroup

7. Node가 삭제 되었는지 확인 - 삭제 될때까지 시간이 걸릴수도 있음

   kubectl get node

FluentBit

1. Namespace 생성

   kubectl create ns logging

2. 로그 수집에 필요한 권한 부여

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: fluent-bit
     namespace: logging
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: fluent-bit-role
   rules:
     - apiGroups: [""]
       resources:
         - namespaces
         - pods
       verbs: ["get", "list", "watch"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: fluent-bit-role-binding
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: fluent-bit-role
   subjects:
     - kind: ServiceAccount
       name: fluent-bit
       namespace: logging
   EOF

3. FluentBit 설정 파일 생성

   cat <<'EOF' | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: fluent-bit
     namespace: logging
   data:
     fluent-bit.conf: |
       [SERVICE]
           Daemon Off
           Flush 1
           Log_Level info
           Parsers_File /fluent-bit/etc/parsers.conf
           HTTP_Server On
           HTTP_Listen 0.0.0.0
           HTTP_Port 2020
           Health_Check On
       [INPUT]
           Name tail
           Path /var/log/containers/*.log
           multiline.parser docker, cri
           Tag kube.*
           Mem_Buf_Limit 5MB
           Skip_Long_Lines On
       [FILTER]
           Name kubernetes
           Match kube.*
           Merge_Log On
           Keep_Log Off
           K8S-Logging.Parser On
           K8S-Logging.Exclude On
       [OUTPUT]
           Name stdout
           Match kube.*
   EOF

4. FluentBit 배포

   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: fluent-bit
     namespace: logging
     labels:
       app: fluent-bit
   spec:
     selector:
       matchLabels:
         app: fluent-bit
     template:
       metadata:
         labels:
           app: fluent-bit
       spec:
         containers:
         - name: fluent-bit
           image: cr.fluentbit.io/fluent/fluent-bit
           command:
           - /fluent-bit/bin/fluent-bit
           args:
           - --workdir=/fluent-bit/etc
           - --config=/fluent-bit/etc/conf/fluent-bit.conf
           volumeMounts:
           - mountPath: /fluent-bit/etc/conf
             name: config
           - mountPath: /var/log
             name: varlog
         volumes:
         - name: config
           configMap:
             defaultMode: 420
             name: fluent-bit
         - name: varlog
           hostPath:
             path: /var/log
         serviceAccountName: fluent-bit
   EOF

5. 생성된 Pod 확인

   kubectl get pods -n logging

6. FluentBit 로그 확인

   kubectl -n logging logs ds/fluent-bit

7. 데모 애플리케이션 배포

   kubectl create deploy nginx --image=nginx --replicas=3

8. 데모 애플리케이션 로그 확인

   kubectl logs deploy/nginx

9. FluentBit에서 위에서 생성한 Pod의 로그를 수집해서 출력되는지 확인

   kubectl -n logging logs ds/fluent-bit | grep nginx

10. Pod에 부여된 Label 확인

    kubectl get pod --show-labels

11. FluentBit 설정 파일 수정

    cat <<'EOF' | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: fluent-bit
      namespace: logging
    data:
      fluent-bit.conf: |
        [SERVICE]
            Daemon Off
            Flush 1
            Log_Level info
            Parsers_File /fluent-bit/etc/parsers.conf
            HTTP_Server On
            HTTP_Listen 0.0.0.0
            HTTP_Port 2020
            Health_Check On
        [INPUT]
            Name tail
            Path /var/log/containers/*_default_*.log
            multiline.parser docker, cri
            Tag kube.*
            Mem_Buf_Limit 5MB
            Skip_Long_Lines On
        [FILTER]
            Name kubernetes
            Match kube.*
            Merge_Log On
            Keep_Log Off
            K8S-Logging.Parser On
            K8S-Logging.Exclude On
        [OUTPUT]
            Name cloudwatch_logs
            Match kube.*
            region ap-northeast-2
            auto_create_group On
            log_group_name /app/log/mycluster
            log_stream_prefix no-label-pods
            log_group_template /app/log/$kubernetes['labels']['app']
            log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
    EOF

12. FluentBit 재생성

    kubectl delete pod -n logging -l app=fluent-bit

13. FluentBit Pod가 재생성 되었는지 확인

    kubectl get pods -n logging

14. FluentBit 로그 확인

    kubectl -n logging logs ds/fluent-bit

15. 데모 애플리케이션에 부하 발생

    {
        kubectl expose deploy nginx --port 80
        kubectl run load-generator --image=busybox:1.28 -l app=load-generator -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://nginx; done"
    }

16. FluentBit 로그 확인

    kubectl -n logging logs ds/fluent-bit

17. FluentBit에 부여한 ServiceAccount에 CloudWatch로 로그 전송을 할수 있는 권한을 부여한 IAM 연동

    {
        export CLUSTER_NAME=$(kubectl get node \
        -o=jsonpath='{.items[0].metadata.labels.alpha\.eksctl\.io\/cluster-name}')
        
        eksctl utils associate-iam-oidc-provider \
        --cluster=$CLUSTER_NAME \
        --approve \
        --region $AWS_REGION 
        
        eksctl create iamserviceaccount \
        --cluster=$CLUSTER_NAME \
        --namespace=logging \
        --name=fluent-bit \
        --attach-policy-arn=arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \
        --override-existing-serviceaccounts \
        --approve \
        --region $AWS_REGION
    }

18. FluentBit에 부여한 ServiceAccount에 IAM 역할 정보가 업데이트 되었는지 확인

    kubectl -n logging get sa fluent-bit -o yaml

19. FluentBit 재생성

    kubectl delete pod -n logging -l app=fluent-bit

20. FluentBit Pod가 재생성 되었는지 확인

    kubectl get pods -n logging

21. FluentBit 로그 확인

    kubectl -n logging logs ds/fluent-bit

22. ServiceAccount에 IAM 역할 연동 후 Pod 명세에 변경된 사항 확인

    kubectl -n logging get pod -l app=fluent-bit -o yaml

23. FluentBit 설정 파일 수정

    cat <<'EOF' | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: fluent-bit
      namespace: logging
    data:
      fluent-bit.conf: |
        [SERVICE]
            Daemon Off
            Flush 1
            Log_Level info
            Parsers_File /fluent-bit/etc/parsers.conf
            HTTP_Server On
            HTTP_Listen 0.0.0.0
            HTTP_Port 2020
            Health_Check On
        [INPUT]
            Name tail
            Path /var/log/containers/*_default_*.log
            multiline.parser docker, cri
            Tag kube.*
            Mem_Buf_Limit 5MB
            Skip_Long_Lines On
        [FILTER]
            Name kubernetes
            Match kube.*
            Merge_Log On
            Keep_Log Off
            K8S-Logging.Parser On
            K8S-Logging.Exclude On
        [OUTPUT]
            Name cloudwatch_logs
            Match kube.*
            region ${AWS_REGION}
            auto_create_group On
            log_group_name /app/log/mycluster
            log_stream_prefix no-label-pods
            log_group_template /app/log/$kubernetes['labels']['app']
            log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
    EOF

24. FluentBit 재생성

    kubectl delete pod -n logging -l app=fluent-bit

25. FluentBit 설정에 명시한 로그 그룹이 생성됐는지 확인

    aws logs describe-log-groups --log-group-name-prefix /app/log

26. 로그 그룹에 위에서 생성한 Pod의 로그가 추가 되는지 확인

    aws logs describe-log-streams \
    --log-group-name /app/log/nginx

27. 위에서 생성한 Pod 로그가 전송된 로그 스트림을 확인하고 환경변수로 지정

    {
        export LOG_STREAM=$(aws logs describe-log-streams \
        --log-group-name /app/log/nginx \
        --query 'logStreams[0].logStreamName' --output text)
    
        echo $LOG_STREAM
    }

28. 위에서 확인한 로그 스트림에 전송된 로그 확인

    aws logs get-log-events \
    --log-group-name /app/log/nginx \
    --log-stream-name $LOG_STREAM --query 'events[*].message'

29. 새로운 데모 애플리케이션 배포

    {
        kubectl create deploy httpd --image=httpd
        kubectl expose deploy httpd --port 80
        kubectl run load-generator-2 --image=busybox:1.28 -l app=load-generator -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://httpd; done"
    }

30. 새로운 로그 그룹이 생성되는지 확인

    aws logs describe-log-groups --log-group-name-prefix /app/log

31. CloudWatch로 전송된 로그 확인

    {
        export LOG_STREAM=$(aws logs describe-log-streams \
        --log-group-name /app/log/httpd \
        --query 'logStreams[0].logStreamName' --output text)
    
        aws logs get-log-events \
        --log-group-name /app/log/httpd \
        --log-stream-name $LOG_STREAM --query 'events[*].message'
    }

32. 생성한 리소스 삭제

    {
        kubectl delete pod -l app=load-generator
        kubectl delete deploy nginx httpd
        kubectl delete clusterrole fluent-bit-role
        kubectl delete clusterrolebinding fluent-bit-role-binding
        
        eksctl delete iamserviceaccount \
        --cluster=$CLUSTER_NAME \
        --namespace=logging \
        --name=fluent-bit \
        --region $AWS_REGION
        
        kubectl delete ns logging
        
        aws logs delete-log-group --log-group-name /app/log/nginx
        aws logs delete-log-group --log-group-name /app/log/httpd
        aws logs delete-log-group --log-group-name /app/log/load-generator
    }

Node Exporter

1. Node Exporter 설치 가이드 - https://prometheus.io/docs/guides/node-exporter
2. Node Exporter GitHub - https://github.com/prometheus/node_exporter
3. Namespace 생성

   kubectl create ns monitoring

4. Node Exporter 설치

   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: node-exporter
     labels:     
       app: node-exporter
     namespace: monitoring
   spec:
     selector:
       matchLabels:
         app: node-exporter
     template:
       metadata:
         labels:         
           app: node-exporter
         annotations:
           prometheus.io/scrape: "true"
           prometheus.io/path: "/metrics"
           prometheus.io/port: "9100"
       spec:
         hostNetwork: true
         hostPID: true
         containers:
           - name: node-exporter
             image: quay.io/prometheus/node-exporter
             args:
               - --path.procfs=/host/proc
               - --path.sysfs=/host/sys
               - --path.rootfs=/host/root
               - --web.listen-address=0.0.0.0:9100
             ports:
               - name: metrics
                 containerPort: 9100
                 protocol: TCP
             volumeMounts:
               - name: proc
                 mountPath: /host/proc
                 readOnly: true
               - name: sys
                 mountPath: /host/sys
                 readOnly: true
               - name: root
                 mountPath: /host/root
                 mountPropagation: HostToContainer
                 readOnly: true
         volumes:
           - name: proc
             hostPath:
               path: /proc
           - name: sys
             hostPath:
               path: /sys
           - name: root
             hostPath:
               path: /
   EOF

5. Node Exporter가 실행중인지 확인

   kubectl -n monitoring get pod -l app=node-exporter

6. Node Exporter가 내보내는 지표 확인

   kubectl run nginx --image=nginx -it --rm --restart=Never \
   -- curl $(kubectl -n monitoring get pod -l app=node-exporter -o=jsonpath="{.items[0].status.podIP}"):9100/metrics

7. Node Exporter 실행옵션 확인

   kubectl -n monitoring exec ds/node-exporter -- node_exporter -h

8. Node Exporter 삭제

   kubectl delete ds node-exporter -n monitoring