실습 | Kubernetes

Kubernetes

Prerequisite

에 명시된 내용을 참고해서 AWS Load Balancer Controller 설치

Installation

https://istio.io/latest/docs/setup/getting-started/#download

Ingress Gateway

TLS

Istio 설치

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
          service:
            externalTrafficPolicy: Cluster
            loadBalancerSourceRanges:
              - 0.0.0.0/0
EOF

CA 인증서 생성

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
-subj '/CN=example.com' -keyout ca-key.pem -out ca-cert.pem

example.com 도메인에 인증서 서명 요청 및 키 생성

openssl req -newkey rsa:2048 -nodes \
-subj "/CN=*.example.com" -keyout example.com.key -out example.com.csr

인증서 서명

openssl x509 -req -sha256 -days 365 -CA ca-cert.pem -CAkey ca-key.pem \
-set_serial 0 -in example.com.csr -out example.com.crt

Secret 생성

kubectl create -n istio-system secret tls example-com \
--key=example.com.key --cert=example.com.crt

Gateway 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: example-com
    hosts:
    - "*"
EOF

VirtualService 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.example.com"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

데모 애플리케이션 배포

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  selector:
    app: nginx
EOF

데모 애플리케이션 호출 - HTTPS

curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v \
https://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Gateway에 HTTPS Redirect 설정

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: example-com
    hosts:
    - "*"
EOF

데모 애플리케이션 호출 - HTTP

curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v -L \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Gateway에 설정한 HTTPS Redirect 삭제

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: example-com
    hosts:
    - "*"
EOF

VirtualService HTTPS Redirect 설정

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.example.com"
  gateways:
  - istio-gateway
  http:
  - match:
    - scheme:
        exact: http
    redirect:
      scheme: https
      redirectCode: 308
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

데모 애플리케이션 호출 - HTTPS

curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v \
https://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

데모 애플리케이션 호출 - HTTP

curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v -L \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

리소스 삭제

kubectl delete deploy nginx
kubectl delete svc nginx
kubectl delete virtualservices.networking.istio.io nginx 
kubectl delete gateways.networking.istio.io istio-gateway
kubectl delete -n istio-system secret example-com
istioctl x uninstall -y --purge

TLS with NLB

AWS Certificate Manager를 통해서 SSL 인증서 발급

사용할 ACM 인증서의 ARN을 환경변수로 지정

export SSL_CERT=<ACM_CERT_ARN>

e.g.

export SSL_CERT=arn:aws:acm:ap-northeast-2:111111111111:certificate/6a427716-ffde-4676-a4f1-02086398df7f

Istio 설치

cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: $SSL_CERT
          service:
            externalTrafficPolicy: Cluster
            loadBalancerSourceRanges:
              - 0.0.0.0/0
          service:
            ports:
              - name: http2
                port: 80
                targetPort: 8080
              - name: status-port
                port: 15020
                targetPort: 15020
          overlays:
            - apiVersion: v1
              kind: Service
              name: istio-ingressgateway
              patches:
                - path: spec.ports[2]
                  value:
                    name: https
                    protocol: TCP
                    port: 443
                    targetPort: 8080
EOF

사용할 Top 레벨 도메인주소를 환경변수로 지정
```
export DOMAIN_NAME=<TLD>
```
e.g.
```
export DOMAIN_NAME=example.com
```

Gateway 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

VirtualService 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

데모 애플리케이션 배포

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  selector:
    app: nginx
EOF

아래의 명령어를 실행해서 나온 결과값과 같은 DNS 레코드 생성

echo nginx.$DOMAIN_NAME=$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME
```
데모 애플리케이션 호출 - HTTP
```
curl http://nginx.$DOMAIN_NAME
```

Gateway에 HTTPS Redirect 설정

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```
데모 애플리케이션 호출 - HTTP
```
curl -L http://nginx.$DOMAIN_NAME -v
```
NLB와 Istio Ingress Gateway는 HTTP 프로토콜로 통신하므로 모든 요청이 HTTPS로 전환됨

Gateway에 설정한 HTTPS Redirect 삭제

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```
데모 애플리케이션 호출 - HTTP
```
curl -L http://nginx.$DOMAIN_NAME -v
```

VirtualService에 HTTPS Redirect 설정

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - scheme:
        exact: http
    redirect:
      scheme: https
      redirectCode: 308
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```
데모 애플리케이션 호출 - HTTP
```
curl -L http://nginx.$DOMAIN_NAME -v
```

VirtualService 설정한 HTTPS Redirect 삭제

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```
데모 애플리케이션 호출 - HTTP
```
curl -L http://nginx.$DOMAIN_NAME -v
```

Istio 설정변경 - NLB(HTTP) -> Istio Ingress Gateway(HTTP), NLB(HTTPS) -> Istio Ingress Gateway(HTTPS)로 라우팅 되도록 구성

cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: $SSL_CERT
          service:
            externalTrafficPolicy: Cluster
            loadBalancerSourceRanges:
              - 0.0.0.0/0
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```
데모 애플리케이션 호출 - HTTP
```
curl -L http://nginx.$DOMAIN_NAME -v
```

Gateway에 443 포트 리스너 추가

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTP
    hosts:
    - "*"
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```

Gateway에 HTTPS Redirect 설정

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTP
    hosts:
    - "*"
EOF

데모 애플리케이션 호출 - HTTPS
```
curl https://nginx.$DOMAIN_NAME -v
```
데모 애플리케이션 호출 - HTTP
```
curl -L http://nginx.$DOMAIN_NAME -v
```

데모 애플리케이션 배포 - https://github.com/youngwjung/flask-echo

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: api
  name: api
spec:
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: youngwjung/flask-echo
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app: api
spec:
  ports:
  - port: 80
  selector:
    app: api
EOF

VirtualService 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - "api.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api
        port:
          number: 80
EOF

아래의 명령어를 실행해서 나온 결과값과 같은 DNS 레코드 생성

echo api.$DOMAIN_NAME=$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

데모 애플리케이션 호출 - HTTPS

curl -v -X POST https://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

데모 애플리케이션 호출 - HTTP

curl -L -v -X POST http://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

Gateway에 설정한 HTTPS Redirect 삭제

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTP
    hosts:
    - "*"
EOF

데모 애플리케이션 호출 - HTTP

curl -v -X POST http://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

VirtualService에 HTTPS Redirect 설정

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - "api.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - port: 80
    redirect:
      scheme: https
      redirectCode: 308
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api
        port:
          number: 80
EOF

데모 애플리케이션 호출 - HTTPS

curl -v -X POST https://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

데모 애플리케이션 호출 - HTTP

curl -v -L -X POST http://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

리소스 삭제

kubectl delete deploy nginx api
kubectl delete svc nginx api
kubectl delete virtualservices.networking.istio.io nginx api
kubectl delete gateways.networking.istio.io istio-gateway
istioctl x uninstall -y --purge

Authorization

Istio 설치

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

Gateway 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

데모 애플리케이션 배포

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  selector:
    app: nginx
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - nginx.example.com
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

데모 애플리케이션 호출

curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

내 컴퓨터(혹은 현재 접속된 Cloud9 인스턴스) IP 주소 확인
```
curl -s ifconfig.io
```

내 컴퓨터에서 Ingress Gateway로 접근 차단

cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: DENY
  rules:
  - from:
    - source:
        ipBlocks: 
        - $(curl -s ifconfig.io)
EOF

데모 애플리케이션 호출

curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Ingress Gateway 로그 확인

kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

Envoy 로그 활성화

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

Ingress Gateway 재생성

kubectl -n istio-system delete pod -l app=istio-ingressgateway

데모 애플리케이션 호출

curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Ingress Gateway 로그 확인

kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

NLB 설정 변경

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

데모 애플리케이션 호출

curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Ingress Gateway 로그 확인

kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

NLB 설정 변경

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

Envoy Filter 생성 - Proxy Protocol

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.filters.listener.proxy_protocol
        - name: envoy.filters.listener.tls_inspector
EOF

Envoy Filter 생성 - X-Forwarded-For 헤더

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingressgateway-settings
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          skip_xff_append: false
          use_remote_address: true
          xff_num_trusted_hops: 1
EOF

데모 애플리케이션 호출

curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Ingress Gateway 로그 확인

kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

AuthorizationPolicy 수정 - https://istio.io/latest/docs/tasks/security/authorization/authz-ingress/#ip-based-allow-list-and-deny-list

cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: DENY
  rules:
  - from:
    - source:
        remoteIpBlocks: 
        - $(curl -s ifconfig.io)
EOF

데모 애플리케이션 호출

curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

Ingress Gateway 로그 확인

kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

리소스 삭제

{
    kubectl delete deploy nginx
    kubectl delete svc nginx
    istioctl x uninstall -y --purge
}

Service Entry

Introduction

Istio 설치

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

default 네임스페이스에 생성되는 Pod에 프록시 주입 설정
```
kubectl label namespace default istio-injection=enabled --overwrite
```
Pod 생성
```
kubectl run nginx --image=nginx
```
Pod가 생성되고 프록시 컨테이너가 추가되었는지 확인
```
kubectl get pod nginx
```
생성한 Pod에서 httpbin.org 접근 시도
```
kubectl exec nginx -- curl -si httpbin.org/json
```

Istio 아웃바운드 트래픽 정책 변경 - https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-OutboundTrafficPolicy-Mode

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

Pod에서 httpbin.org 접근 시도

kubectl exec nginx -- curl -si httpbin.org/json

ServiceEntry 생성 - https://istio.io/latest/docs/reference/config/networking/service-entry/#ServiceEntry

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-org
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
EOF

Pod에서 httpbin.org 접근 시도

kubectl exec nginx -- curl -si httpbin.org/json

리소스 삭제

kubectl delete serviceentries.networking.istio.io httpbin-org
kubectl delete pod nginx
kubectl label namespace default istio-injection-
istioctl x uninstall -y --purge

Deep Dive

Istio 설치

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

default 네임스페이스에 생성되는 Pod에 프록시 주입 설정
```
kubectl label namespace default istio-injection=enabled --overwrite
```
Pod 생성
```
kubectl run nginx --image=nginx
```
Pod가 생성되고 프록시 컨테이너가 추가되었는지 확인
```
kubectl get pod nginx
```
Sidecar Proxy의 리스너 설정 확인 - https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/#deep-dive-into-envoy-configuration
```
istioctl proxy-config listeners nginx
```
Istio Sidecar Proxy가 사용하는 포트 확인 - https://istio.io/latest/docs/ops/deployment/requirements/#ports-used-by-istio
Sidecar Proxy의 아웃바운드 설정 확인 - https://faun.pub/understanding-how-envoy-sidecar-intercept-and-route-traffic-in-istio-service-mesh-20fea2a78833
```
istioctl proxy-config listeners nginx --port 15001 -o json
```
요약: 목적지 포트가 15001일 경우에는 BlackHoleCluster로 아닐 경우에는 PassthroughCluster로
Sidecar Proxy의 클러스터 설정 확인
```
istioctl proxy-config cluster nginx
```
PassthroughCluster의 상세 설정 확인 - InboundPassthroughClusterIpv4 및 PassthroughCluster에 대한 정보가 보여지므로 name 값을 확인해서 PassthroughCluster에 대한 정보만 확인
```
istioctl proxy-config cluster nginx --fqdn PassthroughCluster -o json
```

BlackHoleCluster의 상세 설정 확인

istioctl proxy-config cluster nginx --fqdn BlackHoleCluster -o json

생성한 Pod에서 httpbin.org 접근 시도
```
kubectl exec nginx -- curl -si httpbin.org/json
```

Istio 아웃바운드 트래픽 정책 변경

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

Pod에서 httpbin.org 접근 시도

kubectl exec nginx -- curl -si http://httpbin.org/json

Sidecar Proxy의 리스너 설정 확인
```
istioctl proxy-config listeners nginx
```
Sidecar Proxy의 아웃바운드 설정 확인
```
istioctl proxy-config listeners nginx --port 15001 -o json
```
요약: 목적지 포트가 15001일 경우에는 BlackHoleCluster로 아닐 경우에도 BlackHoleCluster로
BlackHoleCluster의 동작방식 확인 - https://istio.io/latest/blog/2019/monitoring-external-service-traffic/#what-are-blackhole-and-passthrough-clusters

ServiceEntry 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-org
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
EOF

Pod에서 httpbin.org 접근 시도

kubectl exec nginx -- curl -si httpbin.org/json

Sidecar Proxy의 Route 설정 확인
```
istioctl proxy-config route nginx
```
Sidecar Proxy의 Route 설정 확인 - 80/httpbin.org
```
istioctl proxy-config route nginx --name 80 -o json
```
요약: 포트 80으로 나갈때 호스트가 httpbin.org을 경우 httpbin.org 클러스터로 아닐 경우 502 반환

Pod에서 google.com 접근 시도

kubectl exec nginx -- curl -si google.com

Sidecar Proxy의 클러스터 설정 확인
```
istioctl proxy-config cluster nginx
```

httpbin.org 의 상세 설정 확인

istioctl proxy-config cluster nginx --fqdn httpbin.org -o json

리소스 삭제

kubectl delete serviceentries.networking.istio.io httpbin-org
kubectl delete pod nginx
kubectl label namespace default istio-injection-
istioctl x uninstall -y --purge

External HTTPS Proxy

Istio 설치

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

default 네임스페이스에 생성되는 Pod에 프록시 주입 설정
```
kubectl label namespace default istio-injection=enabled --overwrite
```
HTTPS Proxy를 배포할 Namespace 생성
```
kubectl create namespace external
```

Squid proxy 설정파일 생성

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: squid-proxy-conf
  namespace: external
data:
  squid.conf: |
    http_port 3128

    acl SSL_ports port 443
    acl CONNECT method CONNECT
    
    http_access deny CONNECT !SSL_ports
    http_access allow localhost manager
    http_access deny manager
    http_access allow all
    
    coredump_dir /var/spool/squid
EOF

Squid proxy 생성

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: squid
  namespace: external
spec:
  volumes:
  - name: proxy-config
    configMap:
      name: squid-proxy-conf
  containers:
  - name: squid
    image: datadog/squid
    ports:
    - containerPort: 3128
    volumeMounts:
    - name: proxy-config
      mountPath: /etc/squid
EOF

Squid proxy 생성 확인
```
kubectl -n external get pod
```

Sidecar Proxy가 없는 Pod 생성

kubectl -n external run nginx --image=nginx

Squid proxy를 통해서 외부 HTTPS 서비스 호출

kubectl -n external exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

Squid proxy를 통해서 외부 HTTP 서비스 호출

kubectl -n external exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

Squid proxy 로그 확인

kubectl -n external exec squid -- tail /var/log/squid/access.log

Sidecar Proxy를 포함하는 Pod 생성
```
kubectl run nginx --image=nginx
```

Squid proxy를 통해서 외부 HTTPS 서비스 호출

kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

Squid proxy를 통해서 외부 HTTP 서비스 호출

kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

Squid proxy 로그 확인

kubectl -n external exec squid -- tail /var/log/squid/access.log

Sidecar Proxy 로그 확인
```
kubectl logs nginx -c istio-proxy
```

Istio 아웃바운드 트래픽 정책 변경

cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

Squid proxy를 통해서 외부 HTTPS 서비스 호출

kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

Squid proxy를 통해서 외부 HTTP 서비스 호출

kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

Squid proxy 로그 확인

kubectl -n external exec squid -- tail /var/log/squid/access.log

Sidecar Proxy 로그 확인
```
kubectl logs nginx -c istio-proxy
```

ServiceEntry 생성

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-proxy
spec:
  hosts:
  - squid
  addresses:
  - $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}')/32
  ports:
  - number: 3128
    name: http
    protocol: HTTP
  location: MESH_EXTERNAL
EOF

Squid proxy를 통해서 외부 HTTPS 서비스 호출

kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

Squid proxy를 통해서 외부 HTTP 서비스 호출

kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

Squid proxy 로그 확인

kubectl -n external exec squid -- tail /var/log/squid/access.log

Sidecar Proxy 로그 확인
```
kubectl logs nginx -c istio-proxy
```
에러 로그 내용 리뷰

ServiceEntry 변경 - TCP 프로토콜 사용

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-proxy
spec:
  hosts:
  - squid
  addresses:
  - $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}')/32
  ports:
  - number: 3128
    name: tcp
    protocol: TCP
  location: MESH_EXTERNAL
EOF

Squid proxy를 통해서 외부 HTTPS 서비스 호출

kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

Squid proxy를 통해서 외부 HTTP 서비스 호출

kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

Squid proxy 로그 확인

kubectl -n external exec squid -- tail /var/log/squid/access.log

Sidecar Proxy 로그 확인
```
kubectl logs nginx -c istio-proxy
```

리소스 삭제

kubectl delete serviceentries.networking.istio.io external-proxy
kubectl delete pod nginx
kubectl delete namespace external
kubectl label namespace default istio-injection-
istioctl x uninstall -y --purge