실습

Prerequisite

  1. AWS Load Balancer Controller에 명시된 내용을 참고해서 AWS Load Balancer Controller 설치

Installation

  1. https://istio.io/latest/docs/setup/getting-started/#download

Ingress Gateway

TLS

  1. Istio 설치

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
          service:
            externalTrafficPolicy: Cluster
            loadBalancerSourceRanges:
              - 0.0.0.0/0
EOF

  2. CA 인증서 생성

    openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 \
-subj '/CN=example.com' -keyout ca-key.pem -out ca-cert.pem

  3. example.com 도메인에 인증서 서명 요청 및 키 생성

    openssl req -newkey rsa:2048 -nodes \
-subj "/CN=*.example.com" -keyout example.com.key -out example.com.csr

  4. 인증서 서명

    openssl x509 -req -sha256 -days 365 -CA ca-cert.pem -CAkey ca-key.pem \
-set_serial 0 -in example.com.csr -out example.com.crt

  5. Secret 생성

    kubectl create -n istio-system secret tls example-com \
--key=example.com.key --cert=example.com.crt

  6. Gateway 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: example-com
    hosts:
    - "*"
EOF

  7. VirtualService 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.example.com"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

  8. 데모 애플리케이션 배포

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  selector:
    app: nginx
EOF

  9. 데모 애플리케이션 호출 - HTTPS

    curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v \
https://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  10. Gateway에 HTTPS Redirect 설정

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    tls:
      httpsRedirect: true
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: example-com
    hosts:
    - "*"
EOF

  11. 데모 애플리케이션 호출 - HTTP

    curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v -L \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  12. Gateway에 설정한 HTTPS Redirect 삭제

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway 
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      credentialName: example-com
    hosts:
    - "*"
EOF

  13. VirtualService HTTPS Redirect 설정

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.example.com"
  gateways:
  - istio-gateway
  http:
  - match:
    - scheme:
        exact: http
    redirect:
      scheme: https
      redirectCode: 308
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

  14. 데모 애플리케이션 호출 - HTTPS

    curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v \
https://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  15. 데모 애플리케이션 호출 - HTTP

    curl -H "Host: nginx.example.com" --cacert ca-cert.pem --insecure -v -L \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  16. 리소스 삭제

    kubectl delete deploy nginx
kubectl delete svc nginx
kubectl delete virtualservices.networking.istio.io nginx 
kubectl delete gateways.networking.istio.io istio-gateway
kubectl delete -n istio-system secret example-com
istioctl x uninstall -y --purge

TLS with NLB

  1. AWS Certificate Manager를 통해서 SSL 인증서 발급

  2. 사용할 ACM 인증서의 ARN을 환경변수로 지정

    export SSL_CERT=<ACM_CERT_ARN>

    e.g.

    export SSL_CERT=arn:aws:acm:ap-northeast-2:111111111111:certificate/6a427716-ffde-4676-a4f1-02086398df7f

  3. Istio 설치

    cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: $SSL_CERT
          service:
            externalTrafficPolicy: Cluster
            loadBalancerSourceRanges:
              - 0.0.0.0/0
          service:
            ports:
              - name: http2
                port: 80
                targetPort: 8080
              - name: status-port
                port: 15020
                targetPort: 15020
          overlays:
            - apiVersion: v1
              kind: Service
              name: istio-ingressgateway
              patches:
                - path: spec.ports[2]
                  value:
                    name: https
                    protocol: TCP
                    port: 443
                    targetPort: 8080
EOF

  4. 사용할 Top 레벨 도메인주소를 환경변수로 지정

    export DOMAIN_NAME=<TLD>

    e.g.

    export DOMAIN_NAME=example.com

  5. Gateway 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

  6. VirtualService 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

  7. 데모 애플리케이션 배포

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  selector:
    app: nginx
EOF

  8. 아래의 명령어를 실행해서 나온 결과값과 같은 DNS 레코드 생성

    echo nginx.$DOMAIN_NAME=$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  9. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME

  10. 데모 애플리케이션 호출 - HTTP

    curl http://nginx.$DOMAIN_NAME

  11. Gateway에 HTTPS Redirect 설정

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
EOF

  12. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  13. 데모 애플리케이션 호출 - HTTP

    curl -L http://nginx.$DOMAIN_NAME -v

  14. NLB와 Istio Ingress Gateway는 HTTP 프로토콜로 통신하므로 모든 요청이 HTTPS로 전환됨

  15. Gateway에 설정한 HTTPS Redirect 삭제

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

  16. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  17. 데모 애플리케이션 호출 - HTTP

    curl -L http://nginx.$DOMAIN_NAME -v

  18. VirtualService에 HTTPS Redirect 설정

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - scheme:
        exact: http
    redirect:
      scheme: https
      redirectCode: 308
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

  19. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  20. 데모 애플리케이션 호출 - HTTP

    curl -L http://nginx.$DOMAIN_NAME -v

  21. VirtualService 설정한 HTTPS Redirect 삭제

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - "nginx.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

  22. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  23. 데모 애플리케이션 호출 - HTTP

    curl -L http://nginx.$DOMAIN_NAME -v

  24. Istio 설정변경 - NLB(HTTP) -> Istio Ingress Gateway(HTTP), NLB(HTTPS) -> Istio Ingress Gateway(HTTPS)로 라우팅 되도록 구성

    cat <<EOF | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: external
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-attributes: load_balancing.cross_zone.enabled=true
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: $SSL_CERT
          service:
            externalTrafficPolicy: Cluster
            loadBalancerSourceRanges:
              - 0.0.0.0/0
EOF

  25. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  26. 데모 애플리케이션 호출 - HTTP

    curl -L http://nginx.$DOMAIN_NAME -v

  27. Gateway에 443 포트 리스너 추가

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTP
    hosts:
    - "*"
EOF

  28. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  29. Gateway에 HTTPS Redirect 설정

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTP
    hosts:
    - "*"
EOF

  30. 데모 애플리케이션 호출 - HTTPS

    curl https://nginx.$DOMAIN_NAME -v

  31. 데모 애플리케이션 호출 - HTTP

    curl -L http://nginx.$DOMAIN_NAME -v

  32. 데모 애플리케이션 배포 - https://github.com/youngwjung/flask-echo

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: api
  name: api
spec:
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: youngwjung/flask-echo
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: api
  labels:
    app: api
spec:
  ports:
  - port: 80
  selector:
    app: api
EOF

  33. VirtualService 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - "api.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api
        port:
          number: 80
EOF

  34. 아래의 명령어를 실행해서 나온 결과값과 같은 DNS 레코드 생성

    echo api.$DOMAIN_NAME=$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  35. 데모 애플리케이션 호출 - HTTPS

    curl -v -X POST https://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

  36. 데모 애플리케이션 호출 - HTTP

    curl -L -v -X POST http://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

  37. Gateway에 설정한 HTTPS Redirect 삭제

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTP
    hosts:
    - "*"
EOF

  38. 데모 애플리케이션 호출 - HTTP

    curl -v -X POST http://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

  39. VirtualService에 HTTPS Redirect 설정

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api
spec:
  hosts:
  - "api.$DOMAIN_NAME"
  gateways:
  - istio-gateway
  http:
  - match:
    - port: 80
    redirect:
      scheme: https
      redirectCode: 308
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api
        port:
          number: 80
EOF

  40. 데모 애플리케이션 호출 - HTTPS

    curl -v -X POST https://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

  41. 데모 애플리케이션 호출 - HTTP

    curl -v -L -X POST http://api.$DOMAIN_NAME/echo -H 'Content-Type: application/json' -d '{"key": "value"}'

  42. 리소스 삭제

    kubectl delete deploy nginx api
kubectl delete svc nginx api
kubectl delete virtualservices.networking.istio.io nginx api
kubectl delete gateways.networking.istio.io istio-gateway
istioctl x uninstall -y --purge

Authorization

  1. Istio 설치

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

  2. Gateway 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
EOF

  3. 데모 애플리케이션 배포

    cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
  selector:
    app: nginx
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx
spec:
  hosts:
  - nginx.example.com
  gateways:
  - istio-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx
        port:
          number: 80
EOF

  4. 데모 애플리케이션 호출

    curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  5. 내 컴퓨터(혹은 현재 접속된 Cloud9 인스턴스) IP 주소 확인

    curl -s ifconfig.io

  6. 내 컴퓨터에서 Ingress Gateway로 접근 차단

    cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: DENY
  rules:
  - from:
    - source:
        ipBlocks: 
        - $(curl -s ifconfig.io)
EOF

  7. 데모 애플리케이션 호출

    curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  8. Ingress Gateway 로그 확인

    kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

  9. Envoy 로그 활성화

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

  10. Ingress Gateway 재생성

    kubectl -n istio-system delete pod -l app=istio-ingressgateway

  11. 데모 애플리케이션 호출

    curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  12. Ingress Gateway 로그 확인

    kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

  13. NLB 설정 변경

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

  14. 데모 애플리케이션 호출

    curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  15. Ingress Gateway 로그 확인 

    kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

  16. NLB 설정 변경

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        serviceAnnotations:
          service.beta.kubernetes.io/aws-load-balancer-type: external
          service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
          service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    pilot:
      k8s:
        resources:
          requests:
            cpu: 200m
            memory: 500Mi
EOF

  17. Envoy Filter 생성 - Proxy Protocol

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: proxy-protocol
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: LISTENER
    patch:
      operation: MERGE
      value:
        listener_filters:
        - name: envoy.filters.listener.proxy_protocol
        - name: envoy.filters.listener.tls_inspector
EOF

  18. Envoy Filter 생성 - X-Forwarded-For 헤더 

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingressgateway-settings
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          skip_xff_append: false
          use_remote_address: true
          xff_num_trusted_hops: 1
EOF

  19. 데모 애플리케이션 호출

    curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  20. Ingress Gateway 로그 확인 

    kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

  21. AuthorizationPolicy 수정 - https://istio.io/latest/docs/tasks/security/authorization/authz-ingress/#ip-based-allow-list-and-deny-list

    cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: DENY
  rules:
  - from:
    - source:
        remoteIpBlocks: 
        - $(curl -s ifconfig.io)
EOF

  22. 데모 애플리케이션 호출

    curl -H "Host: nginx.example.com" \
http://$(kubectl -n istio-system get svc istio-ingressgateway -o=jsonpath='{.status.loadBalancer.ingress[0].hostname}')

  23. Ingress Gateway 로그 확인 

    kubectl -n istio-system logs deploy/istio-ingressgateway --tail 20

  24. 리소스 삭제

    {
    kubectl delete deploy nginx
    kubectl delete svc nginx
    istioctl x uninstall -y --purge
}

Service Entry

Introduction

  1. Istio 설치

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

  2. default 네임스페이스에 생성되는 Pod에 프록시 주입 설정

    kubectl label namespace default istio-injection=enabled --overwrite

  3. Pod 생성

    kubectl run nginx --image=nginx

  4. Pod가 생성되고 프록시 컨테이너가 추가되었는지 확인

    kubectl get pod nginx

  5. 생성한 Pod에서 httpbin.org 접근 시도

    kubectl exec nginx -- curl -si httpbin.org/json

  6. Istio 아웃바운드 트래픽 정책 변경 - https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-OutboundTrafficPolicy-Mode

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

  7. Pod에서 httpbin.org 접근 시도

    kubectl exec nginx -- curl -si httpbin.org/json

  8. ServiceEntry 생성 - https://istio.io/latest/docs/reference/config/networking/service-entry/#ServiceEntry

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-org
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
EOF

  9. Pod에서 httpbin.org 접근 시도

    kubectl exec nginx -- curl -si httpbin.org/json

  10. 리소스 삭제

    kubectl delete serviceentries.networking.istio.io httpbin-org
kubectl delete pod nginx
kubectl label namespace default istio-injection-
istioctl x uninstall -y --purge

Deep Dive

  1. Istio 설치

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

  2. default 네임스페이스에 생성되는 Pod에 프록시 주입 설정

    kubectl label namespace default istio-injection=enabled --overwrite

  3. Pod 생성

    kubectl run nginx --image=nginx

  4. Pod가 생성되고 프록시 컨테이너가 추가되었는지 확인

    kubectl get pod nginx

  5. Sidecar Proxy의 리스너 설정 확인 - https://istio.io/latest/docs/ops/diagnostic-tools/proxy-cmd/#deep-dive-into-envoy-configuration

    istioctl proxy-config listeners nginx

  6. Istio Sidecar Proxy가 사용하는 포트 확인 - https://istio.io/latest/docs/ops/deployment/requirements/#ports-used-by-istio

  7. Sidecar Proxy의 아웃바운드 설정 확인 - https://faun.pub/understanding-how-envoy-sidecar-intercept-and-route-traffic-in-istio-service-mesh-20fea2a78833

    istioctl proxy-config listeners nginx --port 15001 -o json

    요약: 목적지 포트가 15001일 경우에는 BlackHoleCluster로 아닐 경우에는 PassthroughCluster로

  8. Sidecar Proxy의 클러스터 설정 확인

    istioctl proxy-config cluster nginx

  9. PassthroughCluster의 상세 설정 확인 - InboundPassthroughClusterIpv4 및 PassthroughCluster에 대한 정보가 보여지므로 name 값을 확인해서 PassthroughCluster에 대한 정보만 확인

    istioctl proxy-config cluster nginx --fqdn PassthroughCluster -o json

  10. BlackHoleCluster의 상세 설정 확인

    istioctl proxy-config cluster nginx --fqdn BlackHoleCluster -o json

  11. 생성한 Pod에서 httpbin.org 접근 시도

    kubectl exec nginx -- curl -si httpbin.org/json

  12. Istio 아웃바운드 트래픽 정책 변경

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

  13. Pod에서 httpbin.org 접근 시도

    kubectl exec nginx -- curl -si http://httpbin.org/json

  14. Sidecar Proxy의 리스너 설정 확인

    istioctl proxy-config listeners nginx

  15. Sidecar Proxy의 아웃바운드 설정 확인

    istioctl proxy-config listeners nginx --port 15001 -o json

    요약: 목적지 포트가 15001일 경우에는 BlackHoleCluster로 아닐 경우에도 BlackHoleCluster로

  16. BlackHoleCluster의 동작방식 확인 - https://istio.io/latest/blog/2019/monitoring-external-service-traffic/#what-are-blackhole-and-passthrough-clusters

  17. ServiceEntry 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-org
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
EOF

  18. Pod에서 httpbin.org 접근 시도

    kubectl exec nginx -- curl -si httpbin.org/json

  19. Sidecar Proxy의 Route 설정 확인 

    istioctl proxy-config route nginx

  20. Sidecar Proxy의 Route 설정 확인 - 80/httpbin.org

    istioctl proxy-config route nginx --name 80 -o json

    요약: 포트 80으로 나갈때 호스트가 httpbin.org을 경우 httpbin.org 클러스터로 아닐 경우 502 반환

  21. Pod에서 google.com 접근 시도

    kubectl exec nginx -- curl -si google.com

  22. Sidecar Proxy의 클러스터 설정 확인

    istioctl proxy-config cluster nginx

  23. httpbin.org 의 상세 설정 확인

    istioctl proxy-config cluster nginx --fqdn httpbin.org -o json

  24. 리소스 삭제

    kubectl delete serviceentries.networking.istio.io httpbin-org
kubectl delete pod nginx
kubectl label namespace default istio-injection-
istioctl x uninstall -y --purge

External HTTPS Proxy

  1. Istio 설치

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

  2. default 네임스페이스에 생성되는 Pod에 프록시 주입 설정

    kubectl label namespace default istio-injection=enabled --overwrite

  3. HTTPS Proxy를 배포할 Namespace 생성

    kubectl create namespace external

  4. Squid proxy 설정파일 생성

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: squid-proxy-conf
  namespace: external
data:
  squid.conf: |
    http_port 3128

    acl SSL_ports port 443
    acl CONNECT method CONNECT
    
    http_access deny CONNECT !SSL_ports
    http_access allow localhost manager
    http_access deny manager
    http_access allow all
    
    coredump_dir /var/spool/squid
EOF

  5. Squid proxy 생성

    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: squid
  namespace: external
spec:
  volumes:
  - name: proxy-config
    configMap:
      name: squid-proxy-conf
  containers:
  - name: squid
    image: datadog/squid
    ports:
    - containerPort: 3128
    volumeMounts:
    - name: proxy-config
      mountPath: /etc/squid
EOF

  6. Squid proxy 생성 확인 

    kubectl -n external get pod

  7. Sidecar Proxy가 없는 Pod 생성

    kubectl -n external run nginx --image=nginx

  8. Squid proxy를 통해서 외부 HTTPS 서비스 호출

    kubectl -n external exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

  9. Squid proxy를 통해서 외부 HTTP 서비스 호출

    kubectl -n external exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

  10. Squid proxy 로그 확인

    kubectl -n external exec squid -- tail /var/log/squid/access.log

  11. Sidecar Proxy를 포함하는 Pod 생성

    kubectl run nginx --image=nginx

  12. Squid proxy를 통해서 외부 HTTPS 서비스 호출

    kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

  13. Squid proxy를 통해서 외부 HTTP 서비스 호출

    kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

  14. Squid proxy 로그 확인

    kubectl -n external exec squid -- tail /var/log/squid/access.log

  15. Sidecar Proxy 로그 확인

    kubectl logs nginx -c istio-proxy

  16. Istio 아웃바운드 트래픽 정책 변경

    cat <<'EOF' | istioctl install -y -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
EOF

  17. Squid proxy를 통해서 외부 HTTPS 서비스 호출

    kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

  18. Squid proxy를 통해서 외부 HTTP 서비스 호출

    kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

  19. Squid proxy 로그 확인

    kubectl -n external exec squid -- tail /var/log/squid/access.log

  20. Sidecar Proxy 로그 확인

    kubectl logs nginx -c istio-proxy

  21. ServiceEntry 생성

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-proxy
spec:
  hosts:
  - squid
  addresses:
  - $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}')/32
  ports:
  - number: 3128
    name: http
    protocol: HTTP
  location: MESH_EXTERNAL
EOF

  22. Squid proxy를 통해서 외부 HTTPS 서비스 호출

    kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

  23. Squid proxy를 통해서 외부 HTTP 서비스 호출

    kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

  24. Squid proxy 로그 확인

    kubectl -n external exec squid -- tail /var/log/squid/access.log

  25. Sidecar Proxy 로그 확인

    kubectl logs nginx -c istio-proxy

  26. 에러 로그 내용 리뷰

    1. https://github.com/istio/istio/issues/32494

    2. https://github.com/istio/istio/issues/32494#issuecomment-832238635

    3. https://www.envoyproxy.io/docs/envoy/latest/faq/debugging/why_is_envoy_404ing_connect_requests

  27. ServiceEntry 변경 - TCP 프로토콜 사용

    cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: external-proxy
spec:
  hosts:
  - squid
  addresses:
  - $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}')/32
  ports:
  - number: 3128
    name: tcp
    protocol: TCP
  location: MESH_EXTERNAL
EOF

  28. Squid proxy를 통해서 외부 HTTPS 서비스 호출

    kubectl exec nginx -- \
sh -c "HTTPS_PROXY=$(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 curl -si https://httpbin.org/json"

  29. Squid proxy를 통해서 외부 HTTP 서비스 호출

    kubectl exec nginx -- \
sh -c "curl -x $(kubectl get pod -n external squid -o=jsonpath='{.status.podIP}'):3128 -si http://httpbin.org/json"

  30. Squid proxy 로그 확인

    kubectl -n external exec squid -- tail /var/log/squid/access.log

  31. Sidecar Proxy 로그 확인

    kubectl logs nginx -c istio-proxy

  32. 리소스 삭제

    kubectl delete serviceentries.networking.istio.io external-proxy
kubectl delete pod nginx
kubectl delete namespace external
kubectl label namespace default istio-injection-
istioctl x uninstall -y --purge
PreviousIstio (WIP)NextEnvoy (WIP)

Last updated