环境说明

k8s集群集群版本集群名称网络名称服务网格id东西向网关vip
k8s集群01v1.18.15cluster1network1mesh110.100.10.3
k8s集群02v1.18.15cluster2network2mesh110.100.13.3

集群搭建

略过

各集群部署isito

部署前准备

1. 准备istio & isitoctl

bash
  • 01
  • 02
  • 03
  • 04
  • 05
# 官方下载istio的压缩包
rz istio-1.10.3-linux-amd64.tar.gz
tar -zxf istio-1.10.3-linux-amd64.tar.gz
cd istio-1.10.3
cp bin/istioctl /usr/bin/

2. 生成根证书和密钥

官方默认的命令生成的根证书有效期是10年，ca证书是2年

  • 01
  • 02
openssl x509 -in root-cert.pem -noout -dates
openssl x509 -in ca-cert.pem -noout -dates

需要修改提前修改证书有效期限，此处改为100年

  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
  • 07
  • 08
  • 09
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
vim tools/certs/common.mk 

#------------------------------------------------------------------------
# variables: root CA
ROOTCA_DAYS ?= 36500  # 此处修改的是根证书有效期限
ROOTCA_KEYSZ ?= 4096
ROOTCA_ORG ?= Istio
ROOTCA_CN ?= Root CA
KUBECONFIG ?= $(HOME)/.kube/config
ISTIO_NAMESPACE ?= istio-system
# Additional variables are defined in root-ca.conf target below.

#------------------------------------------------------------------------
# variables: intermediate CA
INTERMEDIATE_DAYS ?= 36500  # 此处修改的是ca证书有效期限
INTERMEDIATE_KEYSZ ?= 4096
INTERMEDIATE_ORG ?= Istio
INTERMEDIATE_CN ?= Intermediate CA
INTERMEDIATE_SAN_DNS ?= istiod.istio-system.svc
# Additional variables are defined in %/intermediate.conf target below.

加入同一个mesh网络的集群都要使用此根证书，只需要执行一次

bash
  • 01
  • 02
  • 03
mkdir -p certs
cd certs
make -f ../tools/certs/Makefile.selfsigned.mk root-ca

3. 生成集群1的证书

  • 01
  • 02
# 证书名中的集群名要一致
make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts

4. 生成集群2的证书

  • 01
make -f ../tools/certs/Makefile.selfsigned.mk cluster2-cacerts

在集群1上部署

1. 创建namespace，创建一个私密 cacerts

  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster1/ca-cert.pem \
      --from-file=cluster1/ca-key.pem \
      --from-file=cluster1/root-cert.pem \
      --from-file=cluster1/cert-chain.pem

2. 为 集群1 设置缺省网络

bash
  • 01
kubectl label namespace istio-system topology.istio.io/network=network1

3. 将 集群1 设为主集群，安装东西向网关

  • 01
istioctl install -f cluster1.yaml -y

cluster1.yaml 文件参数

yaml
  • 001
  • 002
  • 003
  • 004
  • 005
  • 006
  • 007
  • 008
  • 009
  • 010
  • 011
  • 012
  • 013
  • 014
  • 015
  • 016
  • 017
  • 018
  • 019
  • 020
  • 021
  • 022
  • 023
  • 024
  • 025
  • 026
  • 027
  • 028
  • 029
  • 030
  • 031
  • 032
  • 033
  • 034
  • 035
  • 036
  • 037
  • 038
  • 039
  • 040
  • 041
  • 042
  • 043
  • 044
  • 045
  • 046
  • 047
  • 048
  • 049
  • 050
  • 051
  • 052
  • 053
  • 054
  • 055
  • 056
  • 057
  • 058
  • 059
  • 060
  • 061
  • 062
  • 063
  • 064
  • 065
  • 066
  • 067
  • 068
  • 069
  • 070
  • 071
  • 072
  • 073
  • 074
  • 075
  • 076
  • 077
  • 078
  • 079
  • 080
  • 081
  • 082
  • 083
  • 084
  • 085
  • 086
  • 087
  • 088
  • 089
  • 090
  • 091
  • 092
  • 093
  • 094
  • 095
  • 096
  • 097
  • 098
  • 099
  • 100
  • 101
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default
  tag: 1.10.3

  values:
    global:
      # 服务网格id
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      # 缺省网络
      network: network1

    gateways:
      istio-ingressgateway:
        autoscaleEnabled: false
    # istiod
    pilot:
      autoscaleEnabled: false
      replicaCount: 5

  components:
    ingressGateways:
      # eastwest
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          # 缺省网络
          topology.istio.io/network: network1
        enabled: true
        k8s:
          replicaCount: 5
          env:
            # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
            - name: ISTIO_META_ROUTER_MODE
              value: "sni-dnat"
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              # 缺省网络
              value: network1
          service:
            # 东西向网关地址
            externalIPs:
            - 10.100.10.3
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017
      # ingressgateway
      - name: istio-ingressgateway
        enabled: true
        k8s:
          replicaCount: 5
          service:
            # nodeport方式
            type: NodePort
            ports:
            - name: status-port
              port: 15021
              targetPort: 15021
            - name: http2
              port: 80
              targetPort: 8080
              nodePort: 31080
            - name: https
              port: 443
              targetPort: 8443
              nodePort: 31443

  # 开启dns代理
  meshConfig:
    defaultConfig:
      proxyMetadata:
        # Enable basic DNS proxying
        ISTIO_META_DNS_CAPTURE: "true"
        # Enable automatic address allocation, optional
        ISTIO_META_DNS_AUTO_ALLOCATE: "true"

    # 取消默认的跨集群负载平衡
    # 强制管理集群中的本地的流量对于单个服务、特定命名空间下的所有服务和网格中的所有服务
    # https://istio.io/latest/zh/docs/ops/configuration/traffic-management/multicluster/
    serviceSettings:
    - settings:
        clusterLocal: true
      hosts:
      - "*.sre.svc.cluster.local"
      - "*.monitoring.svc.cluster.local"
      - "*.ingress-nginx.svc.cluster.local"

4. 开放 集群1 中的服务

  • 01
kubectl apply -n istio-system -f samples/multicluster/expose-services.yaml

5. 生成集群1API Server 可访问的Secret

bash
  • 01
istioctl x create-remote-secret --name=cluster1 > cluster1-remote-secret.yaml

6.在集群1部署v1版本的helloworld

  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
kubectl create namespace sample
kubectl label namespace sample istio-injection=enabled
kubectl apply -f samples/helloworld/helloworld.yaml -l service=helloworld -n sample
kubectl apply -f samples/sleep/sleep.yaml -n sample
# 部署 V1 版的 HelloWorld
kubectl apply -f samples/helloworld/helloworld.yaml -l version=v1 -n sample

在集群2上部署

1.创建namespace，创建一个私密 cacerts

  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
kubectl create namespace istio-system
kubectl create secret generic cacerts -n istio-system \
      --from-file=cluster2/ca-cert.pem \
      --from-file=cluster2/ca-key.pem \
      --from-file=cluster2/root-cert.pem \
      --from-file=cluster2/cert-chain.pem

2. 为 集群2 设置缺省网络

bash
  • 01
kubectl label namespace istio-system topology.istio.io/network=network2

3. 将 集群2 设为主集群，安装东西向网关

  • 01
istioctl install -f cluster2.yaml -y

cluster2.yaml 文件参数

yaml
  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
  • 07
  • 08
  • 09
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      # 服务网格id
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      # 缺省网络
      network: network2

    gateways:
      istio-ingressgateway:
        autoscaleEnabled: false
    # istiod
    pilot:
      autoscaleEnabled: false
      replicaCount: 5

  # eastwest
  profile: default
  components:
    ingressGateways:
      - name: istio-eastwestgateway
        label:
          istio: eastwestgateway
          app: istio-eastwestgateway
          # 缺省网络
          topology.istio.io/network: network2
        enabled: true
        k8s:
          replicaCount: 5
          env:
            # sni-dnat adds the clusters required for AUTO_PASSTHROUGH mode
            - name: ISTIO_META_ROUTER_MODE
              value: "sni-dnat"
            # traffic through this gateway should be routed inside the network
            - name: ISTIO_META_REQUESTED_NETWORK_VIEW
              # 缺省网络
              value: network2
          service:
            # 东西向网关地址
            externalIPs:
            - 10.100.13.3
            ports:
              - name: status-port
                port: 15021
                targetPort: 15021
              - name: tls
                port: 15443
                targetPort: 15443
              - name: tls-istiod
                port: 15012
                targetPort: 15012
              - name: tls-webhook
                port: 15017
                targetPort: 15017

      - name: istio-ingressgateway
        enabled: true
        k8s:
          replicaCount: 5
          service:
            # nodeport方式
            type: NodePort
            ports:
            - name: status-port
              port: 15021
              targetPort: 15021
            - name: http2
              port: 80
              targetPort: 8080
              nodePort: 31080
            - name: https
              port: 443
              targetPort: 8443
              nodePort: 31443

  # 开启dns代理
  meshConfig:
    defaultConfig:
      proxyMetadata:
        # Enable basic DNS proxying
        ISTIO_META_DNS_CAPTURE: "true"
        # Enable automatic address allocation, optional
        ISTIO_META_DNS_AUTO_ALLOCATE: "true"

    # 取消默认的跨集群负载平衡
    # 强制管理集群中的本地的流量对于单个服务、特定命名空间下的所有服务和网格中的所有服务
    # https://istio.io/latest/zh/docs/ops/configuration/traffic-management/multicluster/
    serviceSettings:
    - settings:
        clusterLocal: true
      hosts:
      - "*.sre.svc.cluster.local"
      - "*.monitoring.svc.cluster.local"
      - "*.ingress-nginx.svc.cluster.local"

4. 开放 集群2 中的服务

  • 01
kubectl apply -n istio-system -f samples/multicluster/expose-services.yaml

5.生成集群2 API Server 可访问的Secret

bash
  • 01
istioctl x create-remote-secret --name=cluster2 > cluster2-remote-secret.yaml

6. 在集群2部署v2版本的helloworld

  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
kubectl create namespace sample
kubectl label namespace sample istio-injection=enabled
kubectl apply -f samples/helloworld/helloworld.yaml -l service=helloworld -n sample
kubectl apply -f samples/sleep/sleep.yaml -n sample
# 部署 V2 版的 HelloWorld
kubectl apply -f samples/helloworld/helloworld.yaml -l version=v2 -n sample

启用端点发现

1. 在集群1中安装上面集群2生成的secret

  • 01
  • 02
# 集群1中操作
kubectl apply -f cluster2-remote-secret.yaml

2. 在集群2中安装上面集群1生成的secret

  • 01
  • 02
# 集群2中操作
kubectl apply -f cluster1-remote-secret.yaml

验证跨集群流量

集群内

  • 01
  • 02
for i in `seq 100`;do curl -s curl helloworld.sample:5000/hello;done
for i in `seq 100`;do kubectl exec -n sample -c sleep "$(kubectl get pod -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -- curl -s helloworld.sample:5000/hello;done

相关命令

  • 01
  • 02
  • 03
  • 04
  • 05
  • 06
  • 07
  • 08
  • 09
  • 10
istioctl ps
istioctl pc ep helloworld-v1-5b75657f75-h2sz2.sample |grep helloworld
istioctl pc ep helloworld-v2-7855866d4f-psknw.sample  |grep helloworld

export INGRESS_HOST=172.17.252.60
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
curl -s -I -HHost:sample-ingress.helloworld.com "http://$INGRESS_HOST:$INGRESS_PORT/hello"

export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT
for i in `seq 1 1000`; do curl -s -o /dev/null http://$GATEWAY_URL/productpage; done

参考文档

https://istio.io/latest/zh/docs/tasks/security/cert-management/plugin-ca-cert/
https://istio.io/latest/zh/docs/setup/install/multicluster/multi-primary_multi-network/
https://istio.io/latest/zh/docs/setup/install/multicluster/verify/
https://zhuanlan.zhihu.com/p/489327602
https://www.bingjie.vip/index.php/archives/103/