실습

Manual Scheduling

  1. Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  nodeName: worker-1
EOF

  2. Pod 상태 확인 

    kubectl get pod nginx

  3. 위의 명령어를 실행하면 아래와 같은 결과를 확인할수 있습니다. 

    NAME         READY   STATUS    RESTARTS   AGE
nginx        0/1     Pending   0          4s

  4. 1분 정도 시간이 지난후에 Pod 상태를 확인 

    kubectl get pod nginx

  5. Kubernetes 공식문서 리뷰 - https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodename

  6. Pod Garbage Collector 코드 리뷰 - https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podgc/gc_controller.go#L212

  7. 노드 목록 확인 

    kubectl get node

  8. spec.nodeName에 위의 명령어의 결과로 나온 첫번째 노드 이름을 넣고 Pod를 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  nodeName: $(kubectl get node -o=jsonpath='{.items[0].metadata.name}')
EOF

  9. Pod 상태 확인 

    kubectl get pod nginx

  10. spec.nodeName을 명시하지 않고 Pod 생성

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: httpd
spec:
  containers:
  - image: httpd
    name: httpd
EOF

  11. spec.nodeName을 명시하지 않고 생성한 Pod에 발생한 Event 확인 

    kubectl describe pod httpd

  12. spec.nodeName을 명시하고 생성한 Pod에 발생한 Event 확인 

    kubectl describe pod nginx

  13. Pod 삭제 

    kubectl delete pod nginx httpd

Node Selector

  1. Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
  nodeSelector:
    env: dev
EOF

  2. Pod 상태 확인 

    kubectl get pod nginx

  3. 위의 명령어를 실행하면 아래와 같은 결과를 확인할수 있습니다. 

    NAME          READY   STATUS    RESTARTS   AGE
nginx         0/1     Pending   0          16s

  4. 아래의 명령어를 사용해서 Pod가 배포되지 않는 이유를 찾으세요. 

    kubectl describe pod nginx

  5. Node에 부여된 Label 확인 

    kubectl get node --show-labels

  6. 두번째 노드에 키(Key)는 env 이고 값(Value)은 dev 인 Label 부여 

    kubectl label node \
$(kubectl get node -o=jsonpath='{.items[1].metadata.name}') env=dev

  7. Node에 Label이 부여되었는지 확인

    kubectl get node --show-labels | grep -E 'env=dev|$'

  8. Pending 상태였던 Pod가 배포됐는지 확인 

    kubectl get pod nginx

  9. Pod에 발생한 Event 확인 

    kubectl describe pod nginx

  10. Node에 부여한 Label 삭제 

    kubectl label node \
$(kubectl get node -o=jsonpath='{.items[1].metadata.name}') env-

  11. Node에 Label이 삭제되었는지 확인

    kubectl get node --show-labels | grep -E 'env=dev|$'

  12. Pod 상태 확인 

    kubectl get pod nginx

  13. Pod 삭제 

    kubectl delete pod nginx

Node Affinity

  1. Node에 부여된 Label 확인 

    kubectl get node --show-labels

  2. 각 Node의 가용영역 확인

    kubectl get nodes --label-columns topology.kubernetes.io/zone

  3. Pod 생성 

    for i in {1..4}
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-${i}
    labels:
      app: nginx
  spec:
    containers:
    - image: nginx
      name: nginx
      resources:
        requests:
          cpu: 0.5
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - ap-northeast-2a
              - ap-northeast-2b
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - ap-northeast-2a
        - weight: 1
          preference:
            matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
              - ap-northeast-2b
EOF
done

  4. Pod가 배포된 노드 확인 

    kubectl get pod -l app=nginx -o wide

  5. Node의 가용영역 확인 

    kubectl get nodes --label-columns topology.kubernetes.io/zone

  6. 각각의 Pod가 어떤 절차에 의해서 각각의 Node에 스케쥴링 되었는지 확인

  7. Pod 삭제 

    kubectl delete pod -l app=nginx

  8. 아래의 YAML 파일에 명시된 spec.affinity가 의미하는 것을 설명하고 Pod 생성 

    for i in {1..4}
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-${i}
    labels:
      app: nginx
  spec:
    containers:
    - image: nginx
      name: nginx
      resources:
        requests:
          cpu: 0.5
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: NotIn
              values:
              - ap-northeast-2a
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: NotIn
              values:
              - ap-northeast-2b
EOF
done

  9. Pod가 배포된 노드 확인 

    kubectl get pod -l app=nginx -o wide

  10. 각각의 Pod가 어떤 절차에 의해서 각각의 Node에 스케쥴링 되었는지 확인

  11. Pod 삭제 

    kubectl delete pod -l app=nginx

Pod Affinity

  1. Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: mod-security
  labels:
    security: enhanced
    app: nginx
spec:
  containers:
  - name: modsecurity
    image: owasp/modsecurity:nginx-alpine
EOF

  2. 아래의 YAML 파일에 명시된 spec.template.spec.affinity가 의미하는 것을 설명하고 Deployment 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: security
                operator: In
                values:
                - enhanced
            topologyKey: kubernetes.io/hostname
EOF

  3. Pod가 배포된 노드 확인 

    kubectl get pod -l app=nginx -o wide

  4. Pod 삭제 

    kubectl delete pod mod-security

  5. mod-security Pod가 삭제되어서 Deployment에 명시한 Affinity 규칙이 더 이상 성립되지 않습니다. 이 경우 어떤일이 발생하는지 설명하세요.

  6. Deployment 삭제 

    kubectl delete deploy nginx

  7. 아래의 YAML 파일에 명시된 spec.template.spec.affinity가 의미하는 것을 설명하고 Deployment 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  labels:
    app: cache
spec:
  selector:
    matchLabels:
      app: cache
  replicas: 4
  template:
    metadata:
      labels:
        app: cache
    spec:
      containers:
      - name: redis-server
        image: redis:3.2-alpine
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - cache
            topologyKey: kubernetes.io/hostname
EOF

  8. Pod가 배포된 노드 확인 

    kubectl get pod -l app=cache -o wide

  9. 아래의 YAML 파일에 명시된 spec.template.spec.affinity가 의미하는 것을 설명하고 Deployment 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - cache
            topologyKey: kubernetes.io/hostname
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx
            topologyKey: kubernetes.io/hostname
EOF

  10. Pod가 배포된 노드 확인 

    kubectl get pod -l 'app in (nginx,cache)' -o wide

  11. 생성된 리소스 삭제 

    kubectl delete deploy -l 'app in (nginx,cache)'

Spread Pod Across Cluster

  1. nodeSelector를 명시한 Pod 생성

    for s in web api queue cache database
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: ${s}
    labels:
      app: myapp
  spec:
    nodeSelector:
      topology.kubernetes.io/zone: ap-northeast-2a
    containers:
    - image: busybox
      name: busybox
      command: ["/bin/sleep"]
      args: ["infinity"]
EOF
done

  2. 각 Pod가 배포된 Node 확인

    kubectl get pod -l app=myapp -o wide -A --sort-by=.spec.nodeName

  3. 각 Node별로 배포된 Pod 갯수 확인

    kubectl describe node | grep -E "(^Name:|^Non-terminated)"

  4. nodeSelector를 명시하지 않은 Pod 생성

    for i in {1..5}
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-${i}
    labels:
      app: nginx
  spec:
    containers:
    - image: nginx
      name: nginx
EOF
done

  5. 각 Pod가 배포된 Node 확인

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  6. 각 Node별로 배포된 Pod 갯수 확인

    kubectl describe node | grep -E "(^Name:|^Non-terminated)"

  7. 위에서 nodeSelector를 명시하지 않고 생성한 Pod 삭제

    kubectl delete pod -l app=nginx

  8. 각 Pod가 배포된 Node 확인

    kubectl get pod -l app=myapp -o wide --sort-by=.spec.nodeName

  9. 각 Node별로 배포된 Pod 갯수 확인

    kubectl describe node | grep -E "(^Name:|^Non-terminated)"

  10. Deployment 생성

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
EOF

  11. Deployment를 통해서 생성된 Pod들이 배포된 Node 확인

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  12. EKS 설정창에서 Scheduler의 로깅을 활성화하면 CloudWatch를 통해서 아래와 같은 구성으로 kube-scheduler가 구동되는 것을 확인 가능

    apiVersion: kubescheduler.config.k8s.io/v1beta1
clientConnection:
  acceptContentTypes: ""
  burst: 100
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /etc/kubernetes/scheduler.conf
  qps: 50
enableContentionProfiling: true
enableProfiling: true
healthzBindAddress: 0.0.0.0:10251
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
  leaseDuration: 15s
  renewDeadline: 10s
  resourceLock: leases
  resourceName: kube-scheduler
  resourceNamespace: kube-system
  retryPeriod: 2s
metricsBindAddress: 0.0.0.0:10251
parallelism: 16
percentageOfNodesToScore: 0
podInitialBackoffSeconds: 1
podMaxBackoffSeconds: 10
profiles:
- pluginConfig:
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      kind: DefaultPreemptionArgs
      minCandidateNodesAbsolute: 100
      minCandidateNodesPercentage: 10
    name: DefaultPreemption
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      hardPodAffinityWeight: 1
      kind: InterPodAffinityArgs
    name: InterPodAffinity
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      kind: NodeAffinityArgs
    name: NodeAffinity
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      kind: NodeResourcesFitArgs
    name: NodeResourcesFit
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      kind: NodeResourcesLeastAllocatedArgs
      resources:
      - name: cpu
        weight: 1
      - name: memory
        weight: 1
    name: NodeResourcesLeastAllocated
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      defaultingType: System
      kind: PodTopologySpreadArgs
    name: PodTopologySpread
  - args:
      apiVersion: kubescheduler.config.k8s.io/v1beta1
      bindTimeoutSeconds: 600
      kind: VolumeBindingArgs
    name: VolumeBinding
  plugins:
    bind:
      enabled:
      - name: DefaultBinder
        weight: 0
    filter:
      enabled:
      - name: NodeUnschedulable
        weight: 0
      - name: NodeName
        weight: 0
      - name: TaintToleration
        weight: 0
      - name: NodeAffinity
        weight: 0
      - name: NodePorts
        weight: 0
      - name: NodeResourcesFit
        weight: 0
      - name: VolumeRestrictions
        weight: 0
      - name: EBSLimits
        weight: 0
      - name: GCEPDLimits
        weight: 0
      - name: NodeVolumeLimits
        weight: 0
      - name: AzureDiskLimits
        weight: 0
      - name: VolumeBinding
        weight: 0
      - name: VolumeZone
        weight: 0
      - name: PodTopologySpread
        weight: 0
      - name: InterPodAffinity
        weight: 0
    permit: {}
    postBind: {}
    postFilter:
      enabled:
      - name: DefaultPreemption
        weight: 0
    preBind:
      enabled:
      - name: VolumeBinding
        weight: 0
    preFilter:
      enabled:
      - name: NodeResourcesFit
        weight: 0
      - name: NodePorts
        weight: 0
      - name: PodTopologySpread
        weight: 0
      - name: InterPodAffinity
        weight: 0
      - name: VolumeBinding
        weight: 0
      - name: NodeAffinity
        weight: 0
    preScore:
      enabled:
      - name: InterPodAffinity
        weight: 0
      - name: PodTopologySpread
        weight: 0
      - name: TaintToleration
        weight: 0
      - name: NodeAffinity
        weight: 0
    queueSort:
      enabled:
      - name: PrioritySort
        weight: 0
    reserve:
      enabled:
      - name: VolumeBinding
        weight: 0
    score:
      enabled:
      - name: NodeResourcesBalancedAllocation
        weight: 1
      - name: ImageLocality
        weight: 1
      - name: InterPodAffinity
        weight: 1
      - name: NodeResourcesLeastAllocated
        weight: 1
      - name: NodeAffinity
        weight: 1
      - name: NodePreferAvoidPods
        weight: 10000
      - name: PodTopologySpread
        weight: 2
      - name: TaintToleration
        weight: 1
  schedulerName: default-scheduler

  13. Pod Topology Spread Constraints 및 Default Pod Topology Spread 리뷰

  14. Deployment 삭제

    kubectl delete deploy nginx

  15. 각 Node별로 배포된 Pod 갯수 확인

    kubectl describe node | grep -E "(^Name:|^Non-terminated)"

  16. podAntiAffinity를 명시하고 Pod 생성

    for i in {1..5}
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-${i}
    labels:
      app: nginx
  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - nginx
            topologyKey: kubernetes.io/hostname
    containers:
    - image: nginx
      name: nginx
EOF
done

  17. 위에서 생성한 Pod들이 배포된 Node 확인

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  18. 위에서 생성한 Pod 삭제

    kubectl delete pod -l app=nginx

  19. topologySpreadConstraints를 명시하고 Pod 생성 

    for i in {1..5}
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-${i}
    labels:
      app: nginx
  spec:
    topologySpreadConstraints:
    - maxSkew: 3
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app: nginx
    containers:
    - image: nginx
      name: nginx
EOF
done

  20. 위에서 생성한 Pod들이 배포된 Node 확인

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  21. 리소스 삭제

    kubectl delete pod -l 'app in (nginx,myapp)'

Requests & Limits

  1. Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    resources:
      requests:
        cpu: 2
EOF

  2. Pod 상태 확인 

    kubectl get pod nginx

  3. 위의 명령어를 실행하면 아래와 같은 결과를 확인할수 있습니다. 

    NAME            READY   STATUS    RESTARTS   AGE
nginx           0/1     Pending   0          8s

  4. 아래의 명령어를 사용해서 Pod가 배포되지 않는 이유를 찾으세요. 

    kubectl describe pod nginx

  5. 노드의 상태의 상세값들을 확인 

    kubectl get node -o=jsonpath='{.items[*].status}' | jq

  6. 위에서 확인한 값들중에서 allocatable.cpu가 무엇을 의미하는지 확인 - https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable

    kubectl explain node.status.allocatable.cpu

  7. 노드별로 할당 가용한 CPU 리소스를 확인 

    kubectl get node -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu

  8. Pending 상태인 Pod를 삭제 

    kubectl delete pod nginx

  9. 노드에 할당 가용한 최대 CPU 값을 spec.container.resources.requests.cpu에 넣고 Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - image: nginx
    name: nginx
    resources:
      limits:
        cpu: $(kubectl get node -o=jsonpath='{.items[0].status.allocatable.cpu}')
EOF

  10. Pod가 배포되는지 확인하고 만약 배포되지 않는다면 그 이유를 확인

    kubectl get pod nginx

  11. Pod 삭제 

    kubectl delete pod nginx

  12. 하나의 노드에 키(Key)는 env 이고 값(Value)은 dev 인 Label 부여 

    kubectl label node \
$(kubectl get node -o=jsonpath='{.items[0].metadata.name}') env=dev

  13. Node에 Label이 부여되었는지 확인

    kubectl get node --show-labels | grep -E 'env=dev|$'

  14. Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: stress-1
  labels:
    app: stress
spec:
  containers:
  - image: lorel/docker-stress-ng
    name: stress-ng
    args:
      - --cpu
      - "2"
  nodeSelector:
    env: dev
---
apiVersion: v1
kind: Pod
metadata:
  name: stress-2
  labels:
    app: stress
spec:
  containers:
  - image: lorel/docker-stress-ng
    name: stress-ng
    args:
      - --cpu
      - "2"
    resources:
      requests:
        cpu: 0.5
      limits:
        cpu: 1
  nodeSelector:
    env: dev
---
apiVersion: v1
kind: Pod
metadata:
  name: stress-3
  labels:
    app: stress
spec:
  containers:
  - image: lorel/docker-stress-ng
    name: stress-ng
    args:
      - --cpu
      - "2"
    resources:
      limits:
        cpu: 1
  nodeSelector:
    env: dev
EOF

  15. Pod가 생성됐는지 확인 

    kubectl get pod -l app=stress

  16. Metric Server 설치 

    kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

  17. Pod들의 CPU 사용량 확인 

    kubectl top pod -l app=stress --use-protocol-buffers

  18. 노드별로 할당 가용한 Memory 리소스 확인 

    kubectl get node \
-o custom-columns=NAME:.metadata.name,ALLOCATABLE_MEMORY:.status.allocatable.memory

  19. Pod 재생성 

    cat <<EOF | kubectl replace --force --grace-period=0 -f -
apiVersion: v1
kind: Pod
metadata:
  name: stress-1
  labels:
    app: stress
spec:
  containers:
  - image: lorel/docker-stress-ng
    name: stress-ng
    args:
      - --vm
      - "1"
      - --vm-bytes
      - "1G"
      - --vm-method 
      - all
  nodeSelector:
    env: dev
---
apiVersion: v1
kind: Pod
metadata:
  name: stress-2
  labels:
    app: stress
spec:
  containers:
  - image: lorel/docker-stress-ng
    name: stress-ng
    args:
      - --vm
      - "1"
      - --vm-bytes
      - "1G"
      - --vm-method 
      - all
    resources:
      requests:
        memory: 200Mi
      limits:
        memory: 1Gi
  nodeSelector:
    env: dev
---
apiVersion: v1
kind: Pod
metadata:
  name: stress-3
  labels:
    app: stress
spec:
  containers:
  - image: lorel/docker-stress-ng
    name: stress-ng
    args:
      - --vm
      - "1"
      - --vm-bytes
      - "1G"
      - --vm-method 
      - all
    resources:
      limits:
        cpu: 0.5
        memory: 1Gi
  nodeSelector:
    env: dev
EOF

  20. Pod가 생성되었는지 확인 

    kubectl get pod -l app=stress

  21. Metrics Server가 60초 마다 지표를 수집하기 때문에 2-3분 정도 대기 - https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md#how-often-metrics-are-scraped

    sleep 180

  22. Pod들의 메모리 사용량 확인

    kubectl top pod -l app=stress --use-protocol-buffers

  23. Pod가 배포된 Node에 발생한 Event 확인 

    kubectl describe node $(kubectl get pod -l app=stress \
-o=jsonpath='{.items[0].spec.nodeName}')

  24. Pod의 상태가 OOMKilled가 될때까지 2-3분간 대기 

    sleep 180

  25. Pod 상태 확인

    kubectl get pod -l app=stress

  26. 상태가 Running이 아닌 Pod가 있으면 원인을 확인

    kubectl get pod -l app=stress \
-o custom-columns=NAME:metadata.name,\
STATUS:status.phase,REASON:status.reason,MESSAGE:status.message

  27. 각 Pod별 QoS 클래스 확인

    kubectl get pod -l app=stress \
-o custom-columns=NAME:.metadata.name,\
STATUS:status.phase,QoS:.status.qosClass

  28. 각 QoS 클래스가 의미하는것을 확인 - https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction

  29. Pod가 배포된 Node에 발생한 Event 확인 

    kubectl describe node $(kubectl get pod -l app=stress \
-o=jsonpath='{.items[0].spec.nodeName}')

  30. 새로운 터미널을 열고 Pod가 배포된 Node의 인스턴스 ID를 확인하고 환경변수로 지정

    {
    export INSTANCE_ID=$(kubectl get node $(kubectl get pod -l app=stress \
    -o=jsonpath='{.items[0].spec.nodeName}') \
    -o jsonpath='{.spec.providerID}{"\n"}' | grep -oE "i-[a-z0-9]+")
    echo $INSTANCE_ID
}

  31. Pod가 배포된 Node로 Session Manager 연결

    aws ssm start-session --target $INSTANCE_ID

  32. 커널 로그에서 OOM 관련된 내용이 있는지 확인 - https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#node-out-of-memory-behavior

    sudo tail -n 1000 /var/log/messages | grep -i oom

  33. Session Manager 종료

    exit

  34. 리소스 삭제

    {
    kubectl delete pod -l app=stress
    kubectl label node env- --all
    kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
}

Taints & Tolerations

  1. Pod 생성 

    for i in {1..10}
do
  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: Pod
  metadata:
    name: nginx-${i}
    labels:
      app: nginx
  spec:
    containers:
    - image: nginx
      name: nginx
EOF
done

  2. Pod가 배포된 Node 확인 

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  3. Node 목록 확인 

    kubectl get node

  4. 첫번째 Node에 Taint 부여 

    kubectl taint nodes $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
status=unstable:PreferNoSchedule

  5. Node에 부여된 Taint 확인 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  6. Pod 재생성 

    for i in {1..10}
do
  kubectl get pod nginx-${i} -o yaml | kubectl replace --force -f -
done

  7. Pod가 배포된 Node 확인 

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  8. 첫번째 Node에 부여한 Taint 삭제 

    kubectl taint nodes $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
status-

  9. Node에 부여된 Taint 확인 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  10. 두번째 Node에 Taint 부여 

    kubectl taint nodes $(kubectl get node -o=jsonpath='{.items[1].metadata.name}') \
status=maintenance:NoSchedule

  11. Node에 부여된 Taint 확인 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  12. Pod 삭제 

    kubectl delete pod -l app=nginx

  13. Deployment 생성 

    kubectl create deploy nginx --image=nginx --replicas=10

  14. 위에서 생성한 Deployment를 통해서 생성된 Pod가 배포된 Node 확인 

    kubectl get pod -l app=nginx -o wide --sort-by=.spec.nodeName

  15. Deployment 재생성 

    cat <<EOF | kubectl replace --force -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 10
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      tolerations:
      - key: status
        operator: Equal
        value: maintenance
        effect: NoSchedule
      containers:
      - name:  nginx
        image: nginx
EOF

  16. 위에서 재생성한 Deployment를 통해서 생성된 Pod가 배포된 Node 확인 

    kubectl get pod -o wide --sort-by=.spec.nodeName

  17. Node에 Taint가 부여되어 있고 PodSpec에 Toleration이 설정되어 있을 경우에 어떤 절차를 통해서 Pod가 Node에 스케쥴링 되는지 확인

  18. 두번째 Node에 부여한 Taint 삭제 

    kubectl taint nodes $(kubectl get node -o=jsonpath='{.items[1].metadata.name}') \
status-

  19. Node에 부여된 Taint 확인 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  20. Deployment 생성 

    kubectl get deploy nginx -o yaml | kubectl replace --force -f -

  21. 위에서 재생성한 Deployment를 통해서 생성된 Pod가 배포된 Node 확인 

    kubectl get pod -o wide --sort-by=.spec.nodeName

  22. 첫번째 Node에 Taint 부여 

    kubectl taint nodes $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
status=upgrading:NoExecute

  23. 첫번째 Node에 Label 부여 

    kubectl label nodes $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
tainted=yes

  24. Node에 부여된 Taint 확인 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  25. Taint가 부여된 Node에 배포되었던 Pod가 삭제되고 다른 Node에 새로운 Pod가 생겼는지 확인 - 리소스가 부족할 경우 Pending 상태가 될수도 있음

    kubectl get pod -o wide --sort-by=.spec.nodeName

  26. nodeSelector를 명시한 Pod 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: emergency
spec:
  containers:
  - name: nginx
    image: nginx
  nodeSelector:
    tainted: "yes"
EOF

  27. Pod가 생성되는지 확인

    kubectl get pod emergency

  28. Pod가 생성되지 않으면 그 이유를 확인

    kubectl describe pod emergency

  29. Pod에 toleration 명시 - 60초 동안만 toleration 유지 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: emergency
spec:
  containers:
  - name: nginx
    image: nginx
  nodeSelector:
    tainted: "yes"
  tolerations:
  - key: status
    operator: Equal
    value: upgrading
    effect: NoExecute
    tolerationSeconds: 60
EOF

  30. Pod가 생성되는지 확인

    kubectl get pod emergency

  31. 60초가 지난 후에도 Pod가 존재하는지 확인

    kubectl get pod emergency

  32. 노드에 부여한 Taint 삭제 

    kubectl taint nodes $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
status-

  33. 생성된 Pod들이 Taint가 삭제된 Node로 옮겨가거나 Pending 상태의 Pod들이 Taint가 삭제된 Node에 생성되는지 확인 

    kubectl get pod -l app=nginx

  34. Deployment 삭제 

    kubectl delete deployment nginx

Descheduler

  1. Cluster Autoscaler 가 실행중인 경우에는 해당 실습이 정상적으로 수행되지 않을수도 있으니 아래의 명령어로 Cluster Autoscaler 정지

    kubectl -n kube-system scale deployment cluster-autoscaler --replicas=0

  2. Deployment 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 6
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
EOF

  3. 생성된 Pod 확인 

    kubectl get pods -l app=nginx -o wide --sort-by=spec.nodeName

  4. 첫번째 Node에 있는 Pod를 다른 Node로 강제 이동

    kubectl drain $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
--pod-selector=app=nginx

  5. 재배포된 Pod 확인 

    kubectl get pods -o wide --sort-by=spec.nodeName

  6. Node에 부여된 Taint 확인 - 3번 명령어에서 명시된 Node에 node.kubernetes.io/unschedulable:NoSchedule Taint가 부여됨 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  7. Taint 제거 

    kubectl uncordon $(kubectl get node -o=jsonpath='{.items[0].metadata.name}')

  8. Node에 부여된 Taint 확인 

    kubectl get nodes -o=custom-columns=NodeName:.metadata.name,\
TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,\
TaintEffect:.spec.taints[*].effect

  9. Descheduler 실행 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: descheduler-policy-configmap
  namespace: kube-system
data:
  policy.yaml: |
    apiVersion: descheduler/v1alpha1
    kind: DeschedulerPolicy
    strategies:
      LowNodeUtilization:
        enabled: true
        params:
          nodeResourceUtilizationThresholds:
            thresholds:
              cpu: 30
              memory: 30
              pods: 50
            targetThresholds:
              cpu: 70
              memory: 70
              pods: 80
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: descheduler-cluster-role
rules:
- apiGroups: [""]
  resources: ["events"]
  verbs: ["create", "update"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list", "delete"]
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]
- apiGroups: ["scheduling.k8s.io"]
  resources: ["priorityclasses"]
  verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: descheduler-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: descheduler-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: descheduler-cluster-role
subjects:
  - name: descheduler-sa
    kind: ServiceAccount
    namespace: kube-system
---
apiVersion: batch/v1
kind: Job
metadata:
  name: descheduler-job
  namespace: kube-system
spec:
  parallelism: 1
  completions: 1
  template:
    metadata:
      name: descheduler-pod
    spec:
      priorityClassName: system-cluster-critical
      containers:
        - name: descheduler
          image: k8s.gcr.io/descheduler/descheduler:v0.21.0
          volumeMounts:
          - mountPath: /policy-dir
            name: policy-volume
          command:
            - "/bin/descheduler"
          args:
            - "--policy-config-file"
            - "/policy-dir/policy.yaml"
            - "--v"
            - "3"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
      restartPolicy: "Never"
      serviceAccountName: descheduler-sa
      volumes:
      - name: policy-volume
        configMap:
          name: descheduler-policy-configmap
EOF

  10. Descheduler Job 상태 확인 

    kubectl get job descheduler-job -n kube-system

  11. Descheduler Job 로그 확인 

    kubectl -n kube-system logs job/descheduler-job

  12. 재배포된 Pod 확인 

    kubectl get pods -o wide --sort-by="{.spec.nodeName}"

  13. Descheduler Job 삭제 

    kubectl -n kube-system delete job descheduler-job

  14. Deployment 삭제 

    kubectl delete deploy nginx

  15. Node에 부여된 Label 확인 

    kubectl get node --show-labels

  16. 하나의 노드에 키(Key)는 env 이고 값(Value)은 test 인 Label 부여 

    kubectl label node $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
env=test

  17. Label이 정상적으로 부여 됐는지 확인 

    kubectl get node --show-labels | grep env=test

  18. Deployment 생성 

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: env
                operator: In
                values:
                - test
      containers:
      - image: nginx
        name: nginx
EOF

  19. 생성된 Pod가 env=test Label을 부여한 노드에 배포 됐는지 확인 

    kubectl get pods -o wide

  20. 15번에서 env=test Label을 부여한 노드에서 해당 Label 삭제 

    kubectl label node $(kubectl get node -o=jsonpath='{.items[0].metadata.name}') \
env-

  21. Label이 정상적으로 삭제 되었는지 확인 

    kubectl get node --show-labels | grep env=test

  22. 기존에 생성된 Pod에 변경사항이 있는지 확인 

    kubectl get pods -o wide

  23. 다른 노드에 키(Key)는 env 이고 값(Value)은 test 인 Label 부여 

    kubectl label node $(kubectl get node -o=jsonpath='{.items[1].metadata.name}') \
env=test

  24. Label이 정상적으로 부여 됐는지 확인 

    kubectl get node --show-labels | grep env=test

  25. 기존에 생성된 Pod에 변경사항이 있는지 확인 

    kubectl get pods -o wide

  26. Descheduler 실행 

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: descheduler-policy-configmap
  namespace: kube-system
data:
  policy.yaml: |
    apiVersion: descheduler/v1alpha1
    kind: DeschedulerPolicy
    strategies:
      RemovePodsViolatingNodeAffinity:
        enabled: true
        params:
          nodeAffinityType:
          - requiredDuringSchedulingIgnoredDuringExecution
---
apiVersion: batch/v1
kind: Job
metadata:
  name: descheduler-job
  namespace: kube-system
spec:
  parallelism: 1
  completions: 1
  template:
    metadata:
      name: descheduler-pod
    spec:
      priorityClassName: system-cluster-critical
      containers:
        - name: descheduler
          image: k8s.gcr.io/descheduler/descheduler:v0.21.0
          volumeMounts:
          - mountPath: /policy-dir
            name: policy-volume
          command:
            - "/bin/descheduler"
          args:
            - "--policy-config-file"
            - "/policy-dir/policy.yaml"
            - "--v"
            - "3"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
      restartPolicy: "Never"
      serviceAccountName: descheduler-sa
      volumes:
      - name: policy-volume
        configMap:
          name: descheduler-policy-configmap
EOF

  27. Descheduler Job 상태 확인 

    kubectl get job descheduler-job -n kube-system

  28. Descheduler Job 로그 확인 

    kubectl -n kube-system logs job/descheduler-job

  29. Pod 상태 확인 

    kubectl get pods -o wide --sort-by="{.spec.nodeName}"

  30. 리소스 삭제 

    {
    kubectl delete job descheduler-job -n kube-system
    kubectl delete sa descheduler-sa -n kube-system
    kubectl delete clusterrole descheduler-cluster-role
    kubectl delete clusterrolebinding descheduler-cluster-role-binding
    kubectl delete deploy nginx
}
PreviousSchedulingNextScaling

Last updated