실습

Manual Scaling

1. Deployment 생성

   kubectl create deployment nginx --image=nginx --replicas=1

2. Pod 갯수 확인

   kubectl get pod -l app=nginx

3. Deployment의 replica를 3개로 수정

   kubectl scale deployment nginx --replicas=3 

4. Pod 갯수 확인

   kubectl get pod -l app=nginx

5. ReplicaSet 확인

   kubectl get rs -l app=nginx

6. Deployment의 replica를 2개로 수정

   kubectl scale deployment nginx --replicas=2

7. Deployment의 replica가 1개라면 replica를 3개로 변경

   kubectl scale deployment nginx --current-replicas=1 --replicas=3 

8. 새로운 Deployment 생성

   kubectl create deployment httpd --image=httpd --replicas=1

9. Deployment 중에서 replica가 1개인 Deployment가 있으면 replica를 3개로 변경

   kubectl scale deployment --current-replicas=1 --replicas=3 --all

10. Pod 갯수 확인

    kubectl get pod -l 'app in (nginx,httpd)'

11. 모든 Deployment의 replica를 5개로 변경

    kubectl scale deployment --replicas=5 --all

12. Pod 갯수 확인

    kubectl get pod -l 'app in (nginx,httpd)'

13. 모든 Deployment 삭제

    kubectl delete deployment -l 'app in (nginx,httpd)'

HPA (Horizontal Pod Autoscaler)

1. 데모 애플리케이션 배포 - https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#run-and-expose-php-apache-server

   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: php-apache
   spec:
     selector:
       matchLabels:
         app: php-apache
     replicas: 1
     template:
       metadata:
         labels:
           app: php-apache
       spec:
         containers:
         - name: php-apache
           image: k8s.gcr.io/hpa-example
           ports:
           - containerPort: 80
           resources:
             limits:
               cpu: 500m
             requests:
               cpu: 200m
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: php-apache
     labels:
       app: php-apache
   spec:
     ports:
     - port: 80
     selector:
       app: php-apache
   EOF

2. Pod의 리소스 사용량 확인

   kubectl top pod -l app=php-apache --use-protocol-buffers

3. Metrics Server 설치 - https://github.com/kubernetes-sigs/metrics-server#kubernetes-metrics-server

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

4. API 서버에 등록된 API 목록 확인

   kubectl get apiservices.apiregistration.k8s.io

5. v1beta1.metrics.k8s.io API의 상세 내용 확인

   kubectl get apiservices.apiregistration.k8s.io v1beta1.metrics.k8s.io -o yaml

6. 모든 Pod의 리소스 사용량 확인

   kubectl top pod -A --use-protocol-buffers

7. 모든 Node의 리소스 사용량 확인

   kubectl top node --use-protocol-buffers

8. Metrics Server 로그 확인

   kubectl -n kube-system logs deploy/metrics-server

9. Metrics Server 로그 레벨 변경

   kubectl -n kube-system patch deployment metrics-server --type=json \
   -p='[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--v=6"}]'

10. Metrics Server 로그 확인 - 새로운 Pod가 뜬 다음에 확인

    kubectl -n kube-system logs deploy/metrics-server

11. kubelet에서 제공하는 지표에 대한 접근 권한을 가진 Pod 생성

    cat <<EOF | kubectl apply -f -
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: kubelet-access
    rules:
    - apiGroups: [""]
      resources: 
        - nodes/stats
        - nodes/metrics
      verbs: ["get"]
    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: kubelet-access
    ---
    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: kubelet-access
      namespace: default
    subjects:
    - kind: ServiceAccount
      name: kubelet-access
      namespace: default
    roleRef:
      kind: ClusterRole
      name: kubelet-access
      apiGroup: rbac.authorization.k8s.io
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: curl
    spec:
      serviceAccountName: kubelet-access
      containers:
      - image: curlimages/curl
        name: curl
        command: ["sleep", "3600"]
        env:
          - name: HOST_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
    EOF

12. 위에서 생성한 Pod에서 Metrics Server가 지표를 수집할때 호출하는 Endpoint 호출 - https://github.com/kubernetes-sigs/metrics-server/blob/4436807eec6b07ea649444529eb3b46ddbbd8914/pkg/scraper/client/resource/client.go#L77

    kubectl exec curl -- \
    sh -c 'curl -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://$HOST_IP:10250/metrics/resource -k'

13. kubelet에서 제공하는 모든 지표 확인

    kubectl exec curl -- \
    sh -c 'curl -s -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://$HOST_IP:10250/stats/summary -k'

14. kubelet에서 제공하는 지표중에서 CPU 및 Memory에 대한 지표만 확인

    kubectl exec curl -- \
    sh -c 'curl -s -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://$HOST_IP:10250/stats/summary?only_cpu_and_memory=true -k'

15. API 서버를 통해서 Metrics Server가 지표를 수집할때 호출하는 Endpoint 호출

    kubectl get --raw /api/v1/nodes/$(kubectl get node -o=jsonpath='{.items[0].metadata.name}')/proxy/metrics/resource

16. Autoscaling (HPA)설정

    kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=5

17. 위에서 생성한 HPA 상태 확인

    kubectl get hpa php-apache

18. 데모 애플리케이션에 부하를 발생시키는 Pod 생성

    kubectl run load-generator --image=busybox:1.28 -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

19. HPA 상태 모니터링

    kubectl get hpa php-apache -w

20. Ctrl+C를 입력해서 HPA 모니터링을 중지하고 실제로 Pod가 생겼는지 확인

    kubectl get pod -l app=php-apache

21. Pod의 리소스 사용량 확인

    kubectl top pod -l app=php-apache --use-protocol-buffers

22. 데모 애플리케이션에 부하를 발생시키는 Pod 삭제

    kubectl delete pod load-generator

23. HPA 상태 모니터링

    kubectl get hpa php-apache -w

24. Ctrl+C를 입력해서 HPA 모니터링을 중지하고 HPA 상세 내용 확인

    kubectl describe hpa php-apache

25. Pod의 리소스 사용량 확인

    kubectl top pod -l app=php-apache --use-protocol-buffers

26. HPA 스케일링 동작방식 확인 - https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#stabilization-window, https://github.com/kubernetes/enhancements/blob/master/keps/sig-autoscaling/853-configurable-hpa-scale-velocity/README.md

27. HPA 상태 확인

    kubectl get hpa php-apache

28. Pod 갯수 확인

    kubectl get pod -l app=php-apache

29. 데모 애플리케이션 및 리소스 삭제

    {
        kubectl delete hpa php-apache
        kubectl delete deployment php-apache
        kubectl delete svc php-apache
        kubectl delete pod curl
        kubectl delete clusterrole kubelet-access
        kubectl delete clusterrolebinding kubelet-access
        kubectl delete sa kubelet-access
    }

Cluster Autoscaler

1. Deployment 생성

   cat <<EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx
     labels:
       app: nginx
   spec:
     replicas: 5
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx
           ports:
           - containerPort: 80
           resources:
             requests:
               cpu: 1
               memory: 1Gi
             limits:
               cpu: 2
               memory: 2Gi
   EOF

2. 생성된 Deployment 및 Pod 확인

   kubectl get deploy,pod -l app=nginx

3. Pending 상태의 Pod가 있다면 아래의 명령어를 통해서 그 이유를 확인

   kubectl describe pod \
   $(kubectl get pod -o=jsonpath='{.items[?(@.status.phase=="Pending")].metadata.name}')

4. Cluster Autoscaler에게 부여할 IAM 정책 JSON 파일 생성

   cat <<EOF > cluster-autoscaler-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "autoscaling:DescribeAutoScalingGroups",
                   "autoscaling:DescribeAutoScalingInstances",
                   "autoscaling:DescribeLaunchConfigurations",
                   "autoscaling:DescribeScalingActivities",
                   "ec2:DescribeImages",
                   "ec2:DescribeInstanceTypes",
                   "ec2:DescribeLaunchTemplateVersions",
                   "ec2:GetInstanceTypesFromInstanceRequirements",
                   "eks:DescribeNodegroup"
               ],
               "Resource": [
                   "*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "autoscaling:SetDesiredCapacity",
                   "autoscaling:TerminateInstanceInAutoScalingGroup"
               ],
               "Resource": [
                   "*"
               ]
           }
       ]
   }
   EOF

5. IAM 정책 생성

   aws iam create-policy \
       --policy-name AmazonEKSClusterAutoscalerPolicy \
       --policy-document file://cluster-autoscaler-policy.json

6. Node에 부여된 Label을 통해서 EKS 클러스터 이름 확인하고 환경변수로 저장

   {
       export CLUSTER_NAME=$(kubectl get node \
       -o=jsonpath='{.items[0].metadata.labels.alpha\.eksctl\.io\/cluster-name}')
       echo $CLUSTER_NAME
   }

7. EKS 클러스터가 생성되어 있는 AWS 계정번호 확인하고 환경변수로 저장

   {
       export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
       echo $ACCOUNT_ID
   }

8. IAM OIDC 제공자 활성화

   eksctl utils associate-iam-oidc-provider --region=ap-northeast-2 \
   --cluster=$CLUSTER_NAME --approve

9. ServiceAccount 생성

   eksctl create iamserviceaccount \
   --cluster=$CLUSTER_NAME \
   --namespace=kube-system \
   --name=cluster-autoscaler \
   --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/AmazonEKSClusterAutoscalerPolicy \
   --override-existing-serviceaccounts \
   --approve

10. ServiceAccount가 생성되었는지 확인

    kubectl -n kube-system describe sa cluster-autoscaler

11. ServiceAccount에 명시된 IAM 역할 이름을 확인하고 환경변수로 지정

    {
       IAM_ROLE_NAME=$(kubectl -n kube-system get sa cluster-autoscaler \
       -o=jsonpath='{.metadata.annotations.eks\.amazonaws\.com\/role-arn}' \
       | grep -oP '(?<=role.).*')
       echo $IAM_ROLE_NAME
    }

12. ServiceAccount와 연동된 IAM 역할에 부여된 IAM 정책 확인

    aws iam list-attached-role-policies --role-name $IAM_ROLE_NAME

13. ServiceAccount와 연동된 IAM 역할에 부여된 신뢰관계 정책 확인

    aws iam get-role --role-name $IAM_ROLE_NAME

14. Cluster Autoscaler 설치

    {
        helm repo add autoscaler https://kubernetes.github.io/autoscaler
        helm install cluster-autoscaler autoscaler/cluster-autoscaler \
        --namespace kube-system \
        --set autoDiscovery.clusterName=$CLUSTER_NAME \
        --set rbac.serviceAccount.create=false \
        --set rbac.serviceAccount.name=cluster-autoscaler \
        --set fullnameOverride=cluster-autoscaler
    }

15. Cluster Autoscaler 로그 확인 - ASG map 확인

    kubectl -n kube-system logs deploy/cluster-autoscaler

16. Cluster Autoscaler 설정값 확인

    kubectl -n kube-system get deploy cluster-autoscaler \
    -o=jsonpath='{.spec.template.spec.containers[*].command}' | jq

17. EKS 노드그룹와 연동된 오토스케일링 그룹 이름을 확인하고 환경변수로 지정

    {
      export ASG_NAME=$(aws autoscaling describe-auto-scaling-groups --query \
      "AutoScalingGroups[? Tags[? (Key=='eks:cluster-name') && Value=='$CLUSTER_NAME']].AutoScalingGroupName" --output text)
      echo $ASG_NAME
    }

18. 오토스케일링 그룹의 인스턴스 현황 확인

    aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG_NAME \
    --query "AutoScalingGroups[0].{MinSize: MinSize, MaxSize: MaxSize, DesiredCapacity: DesiredCapacity}" 

19. 오토스케일링 그룹에 부여된 태그 확인 - https://docs.aws.amazon.com/eks/latest/userguide/autoscaling.html

    aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG_NAME \
    --query "AutoScalingGroups[0].Tags"

20. 설치한 Helm 차트의 기본 변수값 확인

    helm get values cluster-autoscaler -n kube-system --all

21. Cluster Autoscaler 설정값 수정

    helm upgrade cluster-autoscaler autoscaler/cluster-autoscaler \
    --namespace kube-system \
    --set autoDiscovery.clusterName=$CLUSTER_NAME \
    --set rbac.serviceAccount.create=false \
    --set rbac.serviceAccount.name=cluster-autoscaler \
    --set fullnameOverride=cluster-autoscaler \
    --set awsRegion=ap-northeast-2

22. Cluster Autoscaler 설정값 확인

    kubectl -n kube-system get deploy cluster-autoscaler \
    -o=jsonpath='{.spec.template.spec.containers[*].command}' | jq 

23. Cluster Autoscaler 로그 확인 - ASG map 확인

    kubectl -n kube-system logs deploy/cluster-autoscaler

24. Pending 상태였던 Pod가 생성 되었는지 확인

    kubectl get pod -l app=nginx

25. Node 갯수 확인

    kubectl get node

26. 오토스케일링 그룹의 인스턴스 현황 확인

    aws autoscaling describe-auto-scaling-groups --auto-scaling-group-names $ASG_NAME \
    --query "AutoScalingGroups[0].{MinSize: MinSize, MaxSize: MaxSize, DesiredCapacity: DesiredCapacity}" 

27. 오토스케일링 그룹의 활동 로그 확인

    aws autoscaling describe-scaling-activities --auto-scaling-group-name $ASG_NAME

28. Cluster Autoscaler에 의해서 삭제 되지 않는 Node 확인 - https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-types-of-pods-can-prevent-ca-from-removing-a-node

29. Deployment 삭제

    kubectl delete deployment nginx

30. Pod 목록 확인

    kubectl get pod -l app=nginx

31. Node가 삭제 되는지 확인

    kubectl get node

32. Cluster Autoscaler 로그 확인

    kubectl -n kube-system logs deploy/cluster-autoscaler

33. Cluster Autoscaler 스케일 다운 동작방식 리뷰 - https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-does-scale-down-work

34. 리소스 삭제

    rm cluster-autoscaler-policy.json

Karpenter

1. Fargate 실행에 필요한 역할에 부여할 신뢰관계 정책 문서 생성

   cat <<EOF > pod-execution-role-trust-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "eks-fargate-pods.amazonaws.com"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   EOF

2. Fargate 실행에 필요한 역할 생성

   aws iam create-role \
   --role-name AmazonEKSFargatePodExecutionRole \
   --assume-role-policy-document file://"pod-execution-role-trust-policy.json"

3. Fargate 실행에 필요한 역할에 필요한 권한 부여

   aws iam attach-role-policy \
   --policy-arn arn:aws:iam::aws:policy/AmazonEKSFargatePodExecutionRolePolicy \
   --role-name AmazonEKSFargatePodExecutionRole

4. Fargate 프로필 생성

   aws eks create-fargate-profile \
   --fargate-profile-name core-dns \
   --cluster-name $CLUSTER_NAME \
   --pod-execution-role-arn arn:aws:iam::$ACCOUNT_ID:role/AmazonEKSFargatePodExecutionRole \
   --selectors '[{"namespace": "kube-system", "labels": {"k8s-app": "kube-dns"}}]'

5. Fargate 프로필 상태 확인

   aws eks describe-fargate-profile \
   --cluster-name $CLUSTER_NAME \
   --fargate-profile-name core-dns

6. CoreDNS Pod 재생성

   {
       kubectl -n kube-system scale deployment coredns --replicas=0
       kubectl -n kube-system scale deployment coredns --replicas=2
   }

7. CoreDNS Pod 상태 확인

   kubectl -n kube-system get pod -l k8s-app=kube-dns

8. CoreDNS Pod가 생성된 노드 확인

   kubectl -n kube-system get pod -l k8s-app=kube-dns \
   -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

9. Karpenter에 부여할 IAM 정책 문서 생성

   cat <<EOF > karpenter-controller-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Action": [
                   "ssm:GetParameter",
                   "ec2:DescribeImages",
                   "ec2:RunInstances",
                   "ec2:DescribeSubnets",
                   "ec2:DescribeSecurityGroups",
                   "ec2:DescribeLaunchTemplates",
                   "ec2:DescribeInstances",
                   "ec2:DescribeInstanceTypes",
                   "ec2:DescribeInstanceTypeOfferings",
                   "ec2:DescribeAvailabilityZones",
                   "ec2:DeleteLaunchTemplate",
                   "ec2:CreateTags",
                   "ec2:CreateLaunchTemplate",
                   "ec2:CreateFleet",
                   "ec2:DescribeSpotPriceHistory",
                   "pricing:GetProducts"
               ],
               "Effect": "Allow",
               "Resource": "*",
               "Sid": "Karpenter"
           },
           {
               "Action": "ec2:TerminateInstances",
               "Condition": {
                   "StringLike": {
                       "ec2:ResourceTag/karpenter.sh/nodepool": "*"
                   }
               },
               "Effect": "Allow",
               "Resource": "*",
               "Sid": "ConditionalEC2Termination"
           },
           {
               "Effect": "Allow",
               "Action": "iam:PassRole",
               "Resource": "arn:aws:iam::${ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}",
               "Sid": "PassNodeIAMRole"
           },
           {
               "Effect": "Allow",
               "Action": "eks:DescribeCluster",
               "Resource": "arn:aws:eks:${AWS_REGION}:${ACCOUNT_ID}:cluster/${CLUSTER_NAME}",
               "Sid": "EKSClusterEndpointLookup"
           },
           {
               "Sid": "AllowScopedInstanceProfileCreationActions",
               "Effect": "Allow",
               "Resource": "*",
               "Action": [
               "iam:CreateInstanceProfile"
               ],
               "Condition": {
               "StringEquals": {
                   "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
                   "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}"
               },
               "StringLike": {
                   "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
               }
               }
           },
           {
               "Sid": "AllowScopedInstanceProfileTagActions",
               "Effect": "Allow",
               "Resource": "*",
               "Action": [
               "iam:TagInstanceProfile"
               ],
               "Condition": {
               "StringEquals": {
                   "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
                   "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}",
                   "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
                   "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}"
               },
               "StringLike": {
                   "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
                   "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
               }
               }
           },
           {
               "Sid": "AllowScopedInstanceProfileActions",
               "Effect": "Allow",
               "Resource": "*",
               "Action": [
               "iam:AddRoleToInstanceProfile",
               "iam:RemoveRoleFromInstanceProfile",
               "iam:DeleteInstanceProfile"
               ],
               "Condition": {
               "StringEquals": {
                   "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
                   "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}"
               },
               "StringLike": {
                   "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"
               }
               }
           },
           {
               "Sid": "AllowInstanceProfileReadActions",
               "Effect": "Allow",
               "Resource": "*",
               "Action": "iam:GetInstanceProfile"
           }
       ]
   }
   EOF

10. Karpenter에 부여할 IAM 정책 생성

    aws iam create-policy \
    --policy-name KarpenterRole-${CLUSTER_NAME} \
    --policy-document file://karpenter-controller-policy.json

11. Karpenter에 부여할 ServiceAccount 생성

    eksctl create iamserviceaccount \
    --cluster=$CLUSTER_NAME \
    --namespace=karpenter \
    --name=karpenter \
    --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/KarpenterRole-${CLUSTER_NAME} \
    --override-existing-serviceaccounts \
    --approve

12. Fargate 프로필 생성

    aws eks create-fargate-profile \
    --fargate-profile-name karpenter \
    --cluster-name $CLUSTER_NAME \
    --pod-execution-role-arn arn:aws:iam::$ACCOUNT_ID:role/AmazonEKSFargatePodExecutionRole \
    --selectors '[{"namespace": "karpenter"}]'

13. Fargate 프로필 상태 확인

    aws eks describe-fargate-profile \
    --cluster-name $CLUSTER_NAME \
    --fargate-profile-name karpenter

14. Karpenter 설치

    helm install karpenter oci://public.ecr.aws/karpenter/karpenter \
    --namespace karpenter \
    --set settings.clusterName=${CLUSTER_NAME} \
    --set serviceAccount.create=false \
    --set serviceAccount.name=karpenter \
    --set replicas=1

15. Karpenter 생성 확인

    kubectl get pod -n karpenter -o wide

16. Karpenter 로그 확인

    kubectl -n karpenter logs deploy/karpenter

17. 노드에 부여할 IAM 역할 생성

    {
    cat <<EOF > node-trust-policy.json
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }
    EOF
    
    aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
    --assume-role-policy-document file://node-trust-policy.json
    }

18. EKS 노드에 요구되는 필수 권한 부여

    {
        aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
        --policy-arn "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
    
        aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
        --policy-arn "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
    
        aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
        --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
    
        aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \
        --policy-arn "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }

19. 노드에 부여할 역할에 EKS Access Entry 생성

    aws eks create-access-entry \
    --cluster-name $CLUSTER_NAME \
    --type EC2_LINUX \
    --principal-arn arn:aws:iam::$ACCOUNT_ID:role/KarpenterNodeRole-${CLUSTER_NAME}

20. EC2NodeClass 생성

    cat <<EOF | kubectl apply -f -
    apiVersion: karpenter.k8s.aws/v1
    kind: EC2NodeClass
    metadata:
      name: default
    spec:
      amiSelectorTerms: 
      - alias: al2023@latest
      role: KarpenterNodeRole-${CLUSTER_NAME}
      subnetSelectorTerms:
      - tags:
          alpha.eksctl.io/cluster-name: ${CLUSTER_NAME}
          kubernetes.io/role/elb: "1"
      securityGroupSelectorTerms:
      - tags:
          aws:eks:cluster-name: ${CLUSTER_NAME}
      blockDeviceMappings:
      - deviceName: /dev/xvda
        ebs:
          volumeSize: 20Gi
          volumeType: gp3
          encrypted: true
    EOF

21. 생성된 EC2NodeClass 상태 확인

    kubectl describe ec2nodeclasses.karpenter.k8s.aws default

22. NodePool 생성

    cat <<EOF | kubectl apply -f -
    apiVersion: karpenter.sh/v1
    kind: NodePool
    metadata:
      name: default
    spec:
      template:
        spec:
          requirements:
          - key: kubernetes.io/arch
            operator: In
            values: ["amd64"]
          - key: kubernetes.io/os
            operator: In
            values: ["linux"]
          - key: karpenter.sh/capacity-type
            operator: In
            values: ["on-demand", "spot"]
          - key: karpenter.k8s.aws/instance-category
            operator: In
            values: ["t","m","c","r"]
          - key: karpenter.k8s.aws/instance-generation
            operator: Gt
            values: ["2"]
          - key: karpenter.k8s.aws/instance-memory
            operator: Gt
            values: ["2048"]
          nodeClassRef:
            group: karpenter.k8s.aws
            kind: EC2NodeClass
            name: default
      disruption:
        consolidationPolicy: WhenEmptyOrUnderutilized
        consolidateAfter: 1m
    EOF

23. 생성된 NodePool 상태 확인

    kubectl describe nodepools.karpenter.sh default

24. Cluster Autoscaler 비활성화

    kubectl -n kube-system scale deployment cluster-autoscaler --replicas=0

25. 노드그룹 삭제

    eksctl delete nodegroup \
    --cluster=$CLUSTER_NAME \
    --name=nodegroup \
    --disable-eviction \
    --wait

26. Pod 상태 확인

    kubectl get pod -A

27. Node 목록 확인

    kubectl get node

28. Deployment 생성

    cat <<EOF | kubectl apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      replicas: 5
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx
            ports:
            - containerPort: 80
            resources:
              requests:
                cpu: 1
                memory: 1Gi
              limits:
                cpu: 2
                memory: 2Gi
    EOF

29. Pod 상태 확인

    kubectl get pod -l app=nginx

30. Node 목록 확인

    kubectl get node

31. NodeClaim 확인

    kubectl get nodeclaim

32. Karpenter 로그 확인

    kubectl -n karpenter logs deploy/karpenter

33. Cluster Autoscaler 삭제

    {
        helm uninstall cluster-autoscaler -n kube-system
        eksctl delete iamserviceaccount \
        --cluster=$CLUSTER_NAME \
        --namespace=kube-system \
        --name=cluster-autoscaler
    }

34. Deployment 삭제

    kubectl delete deploy nginx

35. Pod 목록 확인

    kubectl get pod -A

36. Karpenter 로그 확인

    kubectl -n karpenter logs deploy/karpenter

37. Node 목록 확인

    kubectl get node

38. NodeClaim 확인

    kubectl get nodeclaim