실습

DaemonSet

현재 생성되어 있는 DaemonSet 확인
```
kubectl get ds -A
```
DaemonSet를 통해서 생성된 Pod들이 배포된 Node 확인
```
kubectl get pod -l 'k8s-app in (kube-proxy, aws-node)' -A -o wide
```

Node 1개 추가

eksctl scale nodegroup --cluster=mycluster --nodes=3 nodegroup

Node가 추가 되었는지 확인 - 생성 될때까지 시간이 걸릴수도 있음
```
kubectl get node
```
DaemonSet를 통해서 생성되는 Pod들이 새롭게 생성된 Node에 배포되었는지 확인
```
kubectl get pod -l 'k8s-app in (kube-proxy, aws-node)' -A -o wide
```

Node 갯수를 원래대로 조정

eksctl scale nodegroup --cluster=mycluster --nodes=2 nodegroup

Node가 삭제 되었는지 확인 - 삭제 될때까지 시간이 걸릴수도 있음
```
kubectl get node
```

FluentBit

Namespace 생성
```
kubectl create ns logging
```

로그 수집에 필요한 권한 부여

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-role
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-role
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: logging
EOF

FluentBit 설정 파일 생성

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
    [OUTPUT]
        Name stdout
        Match kube.*
EOF

FluentBit 배포

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: cr.fluentbit.io/fluent/fluent-bit
        command:
        - /fluent-bit/bin/fluent-bit
        args:
        - --workdir=/fluent-bit/etc
        - --config=/fluent-bit/etc/conf/fluent-bit.conf
        volumeMounts:
        - mountPath: /fluent-bit/etc/conf
          name: config
        - mountPath: /var/log
          name: varlog
      volumes:
      - name: config
        configMap:
          defaultMode: 420
          name: fluent-bit
      - name: varlog
        hostPath:
          path: /var/log
      serviceAccountName: fluent-bit
EOF

생성된 Pod 확인
```
kubectl get pods -n logging
```
FluentBit 로그 확인
```
kubectl -n logging logs ds/fluent-bit
```

데모 애플리케이션 배포

kubectl create deploy nginx --image=nginx --replicas=3

데모 애플리케이션 로그 확인
```
kubectl logs deploy/nginx
```
FluentBit에서 위에서 생성한 Pod의 로그를 수집해서 출력되는지 확인
```
kubectl -n logging logs ds/fluent-bit | grep nginx
```
Pod에 부여된 Label 확인
```
kubectl get pod --show-labels
```

FluentBit 설정 파일 수정

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
    [INPUT]
        Name tail
        Path /var/log/containers/*_default_*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
    [OUTPUT]
        Name cloudwatch_logs
        Match kube.*
        region ap-northeast-2
        auto_create_group On
        log_group_name /app/log/mycluster
        log_stream_prefix no-label-pods
        log_group_template /app/log/$kubernetes['labels']['app']
        log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
EOF

FluentBit 재생성

kubectl delete pod -n logging -l app=fluent-bit

FluentBit Pod가 재생성 되었는지 확인
```
kubectl get pods -n logging
```
FluentBit 로그 확인
```
kubectl -n logging logs ds/fluent-bit
```

데모 애플리케이션에 부하 발생

{
    kubectl expose deploy nginx --port 80
    kubectl run load-generator --image=busybox:1.28 -l app=load-generator -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://nginx; done"
}

FluentBit 로그 확인
```
kubectl -n logging logs ds/fluent-bit
```

FluentBit에 부여한 ServiceAccount에 CloudWatch로 로그 전송을 할수 있는 권한을 부여한 IAM 연동

{
    export CLUSTER_NAME=$(kubectl get node \
    -o=jsonpath='{.items[0].metadata.labels.alpha\.eksctl\.io\/cluster-name}')
    
    eksctl utils associate-iam-oidc-provider \
    --cluster=$CLUSTER_NAME \
    --approve \
    --region $AWS_REGION 
    
    eksctl create iamserviceaccount \
    --cluster=$CLUSTER_NAME \
    --namespace=logging \
    --name=fluent-bit \
    --attach-policy-arn=arn:aws:iam::aws:policy/CloudWatchLogsFullAccess \
    --override-existing-serviceaccounts \
    --approve \
    --region $AWS_REGION
}

FluentBit에 부여한 ServiceAccount에 IAM 역할 정보가 업데이트 되었는지 확인
```
kubectl -n logging get sa fluent-bit -o yaml
```

FluentBit 재생성

kubectl delete pod -n logging -l app=fluent-bit

FluentBit Pod가 재생성 되었는지 확인
```
kubectl get pods -n logging
```
FluentBit 로그 확인
```
kubectl -n logging logs ds/fluent-bit
```
ServiceAccount에 IAM 역할 연동 후 Pod 명세에 변경된 사항 확인
```
kubectl -n logging get pod -l app=fluent-bit -o yaml
```

FluentBit 설정 파일 수정

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On
    [INPUT]
        Name tail
        Path /var/log/containers/*_default_*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
    [OUTPUT]
        Name cloudwatch_logs
        Match kube.*
        region ${AWS_REGION}
        auto_create_group On
        log_group_name /app/log/mycluster
        log_stream_prefix no-label-pods
        log_group_template /app/log/$kubernetes['labels']['app']
        log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
EOF

FluentBit 재생성

kubectl delete pod -n logging -l app=fluent-bit

FluentBit 설정에 명시한 로그 그룹이 생성됐는지 확인
```
aws logs describe-log-groups --log-group-name-prefix /app/log
```
로그 그룹에 위에서 생성한 Pod의 로그가 추가 되는지 확인
```
aws logs describe-log-streams \
--log-group-name /app/log/nginx
```

위에서 생성한 Pod 로그가 전송된 로그 스트림을 확인하고 환경변수로 지정

{
    export LOG_STREAM=$(aws logs describe-log-streams \
    --log-group-name /app/log/nginx \
    --query 'logStreams[0].logStreamName' --output text)

    echo $LOG_STREAM
}

위에서 확인한 로그 스트림에 전송된 로그 확인

aws logs get-log-events \
--log-group-name /app/log/nginx \
--log-stream-name $LOG_STREAM --query 'events[*].message'

새로운 데모 애플리케이션 배포

{
    kubectl create deploy httpd --image=httpd
    kubectl expose deploy httpd --port 80
    kubectl run load-generator-2 --image=busybox:1.28 -l app=load-generator -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://httpd; done"
}

새로운 로그 그룹이 생성되는지 확인

aws logs describe-log-groups --log-group-name-prefix /app/log

CloudWatch로 전송된 로그 확인

{
    export LOG_STREAM=$(aws logs describe-log-streams \
    --log-group-name /app/log/httpd \
    --query 'logStreams[0].logStreamName' --output text)

    aws logs get-log-events \
    --log-group-name /app/log/httpd \
    --log-stream-name $LOG_STREAM --query 'events[*].message'
}

생성한 리소스 삭제

{
    kubectl delete pod -l app=load-generator
    kubectl delete deploy nginx httpd
    kubectl delete clusterrole fluent-bit-role
    kubectl delete clusterrolebinding fluent-bit-role-binding
    
    eksctl delete iamserviceaccount \
    --cluster=$CLUSTER_NAME \
    --namespace=logging \
    --name=fluent-bit \
    --region $AWS_REGION
    
    kubectl delete ns logging
    
    aws logs delete-log-group --log-group-name /app/log/nginx
    aws logs delete-log-group --log-group-name /app/log/httpd
    aws logs delete-log-group --log-group-name /app/log/load-generator
}

Node Exporter

Node Exporter 설치 가이드 - https://prometheus.io/docs/guides/node-exporter
Node Exporter GitHub - https://github.com/prometheus/node_exporter
Namespace 생성
```
kubectl create ns monitoring
```

Node Exporter 설치

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  labels:     
    app: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:         
        app: node-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "9100"
    spec:
      hostNetwork: true
      hostPID: true
      containers:
        - name: node-exporter
          image: quay.io/prometheus/node-exporter
          args:
            - --path.procfs=/host/proc
            - --path.sysfs=/host/sys
            - --path.rootfs=/host/root
            - --web.listen-address=0.0.0.0:9100
          ports:
            - name: metrics
              containerPort: 9100
              protocol: TCP
          volumeMounts:
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: sys
              mountPath: /host/sys
              readOnly: true
            - name: root
              mountPath: /host/root
              mountPropagation: HostToContainer
              readOnly: true
      volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
        - name: root
          hostPath:
            path: /
EOF

Node Exporter가 실행중인지 확인

kubectl -n monitoring get pod -l app=node-exporter

Node Exporter가 내보내는 지표 확인

kubectl run nginx --image=nginx -it --rm --restart=Never \
-- curl $(kubectl -n monitoring get pod -l app=node-exporter -o=jsonpath="{.items[0].status.podIP}"):9100/metrics

Node Exporter 실행옵션 확인

kubectl -n monitoring exec ds/node-exporter -- node_exporter -h

Node Exporter 삭제

kubectl delete ds node-exporter -n monitoring

PreviousDaemonSet NextJob

Last updated 1 year ago

hashtagDaemonSet

hashtagFluentBit

hashtagNode Exporter

DaemonSet

FluentBit

Node Exporter