【Kubernetes资源篇】ingress-nginx最佳实践详解
作者:mmseoamin日期:2023-12-20

文章目录

    • 一、Ingress Controller理论知识
      • 1、Ingress Controller、Ingress简介
      • 2、四层代理与七层代理的区别
      • 3、Ingress Controller中封装Nginx,为什么不直接用Nginx呢?
      • 4、Ingress Controller代理K8S内部Pod流程
      • 二、实践:部署Ingress Controller高可用架构
        • 1、部署Ingress Controller
        • 2、在Node节点上安装并配置Nginx、keepalived
        • 3、测试主备切换
        • 三、实践:创建Ingress规则进行七层转发
          • 1、基于HTTP七层代理转发后端Pod
          • 2、基于HTTPS七层代理转发后端

            一、Ingress Controller理论知识

            Ingress官方中文参考文档:

            1、Ingress Controller、Ingress简介

            Ingress Controller是一个七层负载调度器,常见的七层负载均衡器有nginx、traefik,以我们熟悉的nginx为例,客户端的请求首先会到Ingress Controller七层负载调度器,由七层负载调度器将请求代理到后端的Pod。

            以Nginx举例,客户端请求首先会到Nginx中,由Nginx中的upstream模块将请求代理到后端的服务上,但是K8s场景下,后端Pod的IP地址不是固定的,因此在Pod前面需要添加一个service资源,请求到达Service,由Service代理到后端的Pod。

            【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-bjf2hcl2-1690102684594)(D:\MD归档文档\IMG\image-20230722173017074.png)],第1张

            Ingress是K8S中的资源,简单理解就是Ingress Controller的配置文件,创建ingress规则在管理Ingress Controller。

            2、四层代理与七层代理的区别

            四层代理:

            • 工作在传输层,可以解析传输层协议,TCP、UDP等。
            • 四层代理 基于传IP+端口方式进行转发。

              七层代理:

              • 工作在应有层,可以解析应用层协议,如HTTP、FTP等。
              • 七层负载工作在四层的基础之上,基于虚拟主机的URL或主机的IP进行转发。

                总体而言,四层代理更关注于网络层面的流量控制和安全,主要基于传输层的信息进行处理;而七层代理更加智能,能够理解和处理应用层协议的内容,提供更加精细的控制和调度。选择使用哪种类型的代理取决于具体需求和使用场景。

                OSI七层模型:

                【Kubernetes资源篇】ingress-nginx最佳实践详解,在这里插入图片描述,第2张

                722190254323.png)]

                3、Ingress Controller中封装Nginx,为什么不直接用Nginx呢?

                在宿主机安装Nginx,只要配置文件有改动,就必须手动reload加载才可以生效,但是如果使用Ingress Controller封装的Nginx,你ingress维护配置,ingress创建好了之后,会自动把配置文件传到Ingress Controller这个Pod中,自动进行reload加载。

                4、Ingress Controller代理K8S内部Pod流程

                第一步:部署Ingress Controller

                第二步:创建Pod,可以使用控制器进行创建

                第三步:创建Service,管理Pod

                第四步:创建Ingress http或https规则

                第五步:测试,客户端通过七层访问

                二、实践:部署Ingress Controller高可用架构

                高可用架构请求转发图:

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-cUwOluxz-1690102684596)(D:\MD归档文档\IMG\image-20230723165049687.png)],第3张

                ingress-nginx GitHub地址:

                ingress-nginx YAML GitHub地址:

                1、部署Ingress Controller

                1、编写YAML文件,基于官方下载,根基自己需求进行对应修改。

                cat ingress-controller-nginx.yaml
                ---
                apiVersion: v1
                kind: Namespace
                metadata:
                  name: ingress-nginx
                  labels:
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                ---
                # Source: ingress-nginx/templates/controller-serviceaccount.yaml
                apiVersion: v1
                kind: ServiceAccount
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx
                  namespace: ingress-nginx
                automountServiceAccountToken: true
                ---
                # Source: ingress-nginx/templates/controller-configmap.yaml
                apiVersion: v1
                kind: ConfigMap
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx-controller
                  namespace: ingress-nginx
                data:
                  allow-snippet-annotations: "true"
                ---
                # Source: ingress-nginx/templates/clusterrole.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRole
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                  name: ingress-nginx
                rules:
                  - apiGroups:
                      - ""
                    resources:
                      - configmaps
                      - endpoints
                      - nodes
                      - pods
                      - secrets
                      - namespaces
                    verbs:
                      - list
                      - watch
                  - apiGroups:
                      - ""
                    resources:
                      - nodes
                    verbs:
                      - get
                  - apiGroups:
                      - ""
                    resources:
                      - services
                    verbs:
                      - get
                      - list
                      - watch
                  - apiGroups:
                      - networking.k8s.io
                    resources:
                      - ingresses
                    verbs:
                      - get
                      - list
                      - watch
                  - apiGroups:
                      - ""
                    resources:
                      - events
                    verbs:
                      - create
                      - patch
                  - apiGroups:
                      - networking.k8s.io
                    resources:
                      - ingresses/status
                    verbs:
                      - update
                  - apiGroups:
                      - networking.k8s.io
                    resources:
                      - ingressclasses
                    verbs:
                      - get
                      - list
                      - watch
                ---
                # Source: ingress-nginx/templates/clusterrolebinding.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRoleBinding
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                  name: ingress-nginx
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: ClusterRole
                  name: ingress-nginx
                subjects:
                  - kind: ServiceAccount
                    name: ingress-nginx
                    namespace: ingress-nginx
                ---
                # Source: ingress-nginx/templates/controller-role.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: Role
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx
                  namespace: ingress-nginx
                rules:
                  - apiGroups:
                      - ""
                    resources:
                      - namespaces
                    verbs:
                      - get
                  - apiGroups:
                      - ""
                    resources:
                      - configmaps
                      - pods
                      - secrets
                      - endpoints
                    verbs:
                      - get
                      - list
                      - watch
                  - apiGroups:
                      - ""
                    resources:
                      - services
                    verbs:
                      - get
                      - list
                      - watch
                  - apiGroups:
                      - networking.k8s.io
                    resources:
                      - ingresses
                    verbs:
                      - get
                      - list
                      - watch
                  - apiGroups:
                      - networking.k8s.io
                    resources:
                      - ingresses/status
                    verbs:
                      - update
                  - apiGroups:
                      - networking.k8s.io
                    resources:
                      - ingressclasses
                    verbs:
                      - get
                      - list
                      - watch
                  - apiGroups:
                      - ""
                    resources:
                      - configmaps
                    resourceNames:
                      - ingress-controller-leader
                    verbs:
                      - get
                      - update
                  - apiGroups:
                      - ""
                    resources:
                      - configmaps
                    verbs:
                      - create
                  - apiGroups:
                      - ""
                    resources:
                      - events
                    verbs:
                      - create
                      - patch
                ---
                # Source: ingress-nginx/templates/controller-rolebinding.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: RoleBinding
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx
                  namespace: ingress-nginx
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: Role
                  name: ingress-nginx
                subjects:
                  - kind: ServiceAccount
                    name: ingress-nginx
                    namespace: ingress-nginx
                ---
                # Source: ingress-nginx/templates/controller-service-webhook.yaml
                apiVersion: v1
                kind: Service
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx-controller-admission
                  namespace: ingress-nginx
                spec:
                  type: ClusterIP
                  ports:
                    - name: https-webhook
                      port: 443
                      targetPort: webhook
                      appProtocol: https
                  selector:
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/component: controller
                ---
                # Source: ingress-nginx/templates/controller-service.yaml
                apiVersion: v1
                kind: Service
                metadata:
                  annotations:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx-controller
                  namespace: ingress-nginx
                spec:
                  type: NodePort
                  ipFamilyPolicy: SingleStack
                  ipFamilies:
                    - IPv4
                  ports:
                    - name: http
                      port: 80
                      protocol: TCP
                      targetPort: http
                      appProtocol: http
                    - name: https
                      port: 443
                      protocol: TCP
                      targetPort: https
                      appProtocol: https
                  selector:
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/component: controller
                ---
                # Source: ingress-nginx/templates/controller-deployment.yaml
                apiVersion: apps/v1
                kind: Deployment
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: ingress-nginx-controller
                  namespace: ingress-nginx
                spec:
                  replicas: 2
                  selector:
                    matchLabels:
                      app.kubernetes.io/name: ingress-nginx
                      app.kubernetes.io/instance: ingress-nginx
                      app.kubernetes.io/component: controller
                  revisionHistoryLimit: 10
                  minReadySeconds: 0
                  template:
                    metadata:
                      labels:
                        app.kubernetes.io/name: ingress-nginx
                        app.kubernetes.io/instance: ingress-nginx
                        app.kubernetes.io/component: controller
                    spec:
                      hostNetwork: true
                      affinity:
                        podAntiAffinity:
                          preferredDuringSchedulingIgnoredDuringExecution:
                            - weight: 100
                              podAffinityTerm:
                                labelSelector:
                                  matchLabels:
                                    app.kubernetes.io/name: ingress-nginx
                                topologyKey: kubernetes.io/hostname
                      dnsPolicy: ClusterFirstWithHostNet
                      containers:
                        - name: controller
                          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.1.0
                          imagePullPolicy: IfNotPresent
                          lifecycle:
                            preStop:
                              exec:
                                command:
                                  - /wait-shutdown
                          args:
                            - /nginx-ingress-controller
                            - --election-id=ingress-controller-leader
                            - --controller-class=k8s.io/ingress-nginx
                            - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
                            - --validating-webhook=:8443
                            - --validating-webhook-certificate=/usr/local/certificates/cert
                            - --validating-webhook-key=/usr/local/certificates/key
                          securityContext:
                            capabilities:
                              drop:
                                - ALL
                              add:
                                - NET_BIND_SERVICE
                            runAsUser: 101
                            allowPrivilegeEscalation: true
                          env:
                            - name: POD_NAME
                              valueFrom:
                                fieldRef:
                                  fieldPath: metadata.name
                            - name: POD_NAMESPACE
                              valueFrom:
                                fieldRef:
                                  fieldPath: metadata.namespace
                            - name: LD_PRELOAD
                              value: /usr/local/lib/libmimalloc.so
                          livenessProbe:
                            failureThreshold: 5
                            httpGet:
                              path: /healthz
                              port: 10254
                              scheme: HTTP
                            initialDelaySeconds: 10
                            periodSeconds: 10
                            successThreshold: 1
                            timeoutSeconds: 1
                          readinessProbe:
                            failureThreshold: 3
                            httpGet:
                              path: /healthz
                              port: 10254
                              scheme: HTTP
                            initialDelaySeconds: 10
                            periodSeconds: 10
                            successThreshold: 1
                            timeoutSeconds: 1
                          ports:
                            - name: http
                              containerPort: 80
                              protocol: TCP
                            - name: https
                              containerPort: 443
                              protocol: TCP
                            - name: webhook
                              containerPort: 8443
                              protocol: TCP
                          volumeMounts:
                            - name: webhook-cert
                              mountPath: /usr/local/certificates/
                              readOnly: true
                          resources:
                            requests:
                              cpu: 100m
                              memory: 90Mi
                      nodeSelector:
                        kubernetes.io/os: linux
                      serviceAccountName: ingress-nginx
                      terminationGracePeriodSeconds: 300
                      volumes:
                        - name: webhook-cert
                          secret:
                            secretName: ingress-nginx-admission
                ---
                # Source: ingress-nginx/templates/controller-ingressclass.yaml
                # We don't support namespaced ingressClass yet
                # So a ClusterRole and a ClusterRoleBinding is required
                apiVersion: networking.k8s.io/v1
                kind: IngressClass
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: controller
                  name: nginx
                  namespace: ingress-nginx
                spec:
                  controller: k8s.io/ingress-nginx
                ---
                # Source: ingress-nginx/templates/admission-webhooks/validating-webhook.yaml
                # before changing this value, check the required kubernetes version
                # https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites
                apiVersion: admissionregistration.k8s.io/v1
                kind: ValidatingWebhookConfiguration
                metadata:
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                  name: ingress-nginx-admission
                webhooks:
                  - name: validate.nginx.ingress.kubernetes.io
                    matchPolicy: Equivalent
                    rules:
                      - apiGroups:
                          - networking.k8s.io
                        apiVersions:
                          - v1
                        operations:
                          - CREATE
                          - UPDATE
                        resources:
                          - ingresses
                    failurePolicy: Fail
                    sideEffects: None
                    admissionReviewVersions:
                      - v1
                    clientConfig:
                      service:
                        namespace: ingress-nginx
                        name: ingress-nginx-controller-admission
                        path: /networking/v1/ingresses
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/serviceaccount.yaml
                apiVersion: v1
                kind: ServiceAccount
                metadata:
                  name: ingress-nginx-admission
                  namespace: ingress-nginx
                  annotations:
                    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrole.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRole
                metadata:
                  name: ingress-nginx-admission
                  annotations:
                    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                rules:
                  - apiGroups:
                      - admissionregistration.k8s.io
                    resources:
                      - validatingwebhookconfigurations
                    verbs:
                      - get
                      - update
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/clusterrolebinding.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRoleBinding
                metadata:
                  name: ingress-nginx-admission
                  annotations:
                    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: ClusterRole
                  name: ingress-nginx-admission
                subjects:
                  - kind: ServiceAccount
                    name: ingress-nginx-admission
                    namespace: ingress-nginx
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/role.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: Role
                metadata:
                  name: ingress-nginx-admission
                  namespace: ingress-nginx
                  annotations:
                    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                rules:
                  - apiGroups:
                      - ""
                    resources:
                      - secrets
                    verbs:
                      - get
                      - create
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/rolebinding.yaml
                apiVersion: rbac.authorization.k8s.io/v1
                kind: RoleBinding
                metadata:
                  name: ingress-nginx-admission
                  namespace: ingress-nginx
                  annotations:
                    helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                roleRef:
                  apiGroup: rbac.authorization.k8s.io
                  kind: Role
                  name: ingress-nginx-admission
                subjects:
                  - kind: ServiceAccount
                    name: ingress-nginx-admission
                    namespace: ingress-nginx
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
                apiVersion: batch/v1
                kind: Job
                metadata:
                  name: ingress-nginx-admission-create
                  namespace: ingress-nginx
                  annotations:
                    helm.sh/hook: pre-install,pre-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                spec:
                  template:
                    metadata:
                      name: ingress-nginx-admission-create
                      labels:
                        helm.sh/chart: ingress-nginx-4.0.10
                        app.kubernetes.io/name: ingress-nginx
                        app.kubernetes.io/instance: ingress-nginx
                        app.kubernetes.io/version: 1.1.0
                        app.kubernetes.io/managed-by: Helm
                        app.kubernetes.io/component: admission-webhook
                    spec:
                      containers:
                        - name: create
                          image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
                          imagePullPolicy: IfNotPresent
                          args:
                            - create
                            - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
                            - --namespace=$(POD_NAMESPACE)
                            - --secret-name=ingress-nginx-admission
                          env:
                            - name: POD_NAMESPACE
                              valueFrom:
                                fieldRef:
                                  fieldPath: metadata.namespace
                          securityContext:
                            allowPrivilegeEscalation: false
                      restartPolicy: OnFailure
                      serviceAccountName: ingress-nginx-admission
                      nodeSelector:
                        kubernetes.io/os: linux
                      securityContext:
                        runAsNonRoot: true
                        runAsUser: 2000
                ---
                # Source: ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
                apiVersion: batch/v1
                kind: Job
                metadata:
                  name: ingress-nginx-admission-patch
                  namespace: ingress-nginx
                  annotations:
                    helm.sh/hook: post-install,post-upgrade
                    helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded
                  labels:
                    helm.sh/chart: ingress-nginx-4.0.10
                    app.kubernetes.io/name: ingress-nginx
                    app.kubernetes.io/instance: ingress-nginx
                    app.kubernetes.io/version: 1.1.0
                    app.kubernetes.io/managed-by: Helm
                    app.kubernetes.io/component: admission-webhook
                spec:
                  template:
                    metadata:
                      name: ingress-nginx-admission-patch
                      labels:
                        helm.sh/chart: ingress-nginx-4.0.10
                        app.kubernetes.io/name: ingress-nginx
                        app.kubernetes.io/instance: ingress-nginx
                        app.kubernetes.io/version: 1.1.0
                        app.kubernetes.io/managed-by: Helm
                        app.kubernetes.io/component: admission-webhook
                    spec:
                      containers:
                        - name: patch
                          image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v1.1.1
                          imagePullPolicy: IfNotPresent
                          args:
                            - patch
                            - --webhook-name=ingress-nginx-admission
                            - --namespace=$(POD_NAMESPACE)
                            - --patch-mutating=false
                            - --secret-name=ingress-nginx-admission
                            - --patch-failure-policy=Fail
                          env:
                            - name: POD_NAMESPACE
                              valueFrom:
                                fieldRef:
                                  fieldPath: metadata.namespace
                          securityContext:
                            allowPrivilegeEscalation: false
                      restartPolicy: OnFailure
                      serviceAccountName: ingress-nginx-admission
                      nodeSelector:
                        kubernetes.io/os: linux
                      securityContext:
                        runAsNonRoot: true
                        runAsUser: 2000
                

                2、执行YAML文件

                kubectl apply -f ingress-controller-nginx.yaml
                

                如果执行YAML文件有报错,如下:

                报错内容:Error from server (InternalError): error when creating “ingress.yaml“: Internal error occurred: fail

                报错解决方法:

                kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
                

                3、查看创建的Pod资源状态是否已运行

                kubectl get pod -n ingress-nginx
                

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-bODP7iYg-1690102684603)(D:\MD归档文档\IMG\image-20230723131517381.png)],第4张

                2、在Node节点上安装并配置Nginx、keepalived

                1、上面部署ingress controller分配在不同的两台Node节点(两台Node节点同步操作)

                yum install  epel-release  nginx keepalived nginx-mod-stream nc -y
                

                2、修改 nginx.conf 配置文件(两台Node节点同步操作)

                mv /etc/nginx/nginx.conf{,.$(date +%F)}
                vim /etc/nginx/nginx.conf
                user nginx;
                worker_processes auto;
                error_log /var/log/nginx/error.log;
                pid /run/nginx.pid;
                include /usr/share/nginx/modules/*.conf;
                events {
                    worker_connections 1024;
                }
                # 四层负载
                stream {
                    log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
                    access_log  /var/log/nginx/k8s-access.log  main;
                	# 定义后端负载节点
                    upstream k8s-ingress-controller {
                       server 16.32.15.201:80 weight=5 max_fails=3 fail_timeout=30s;
                       server 16.32.15.202:80 weight=5 max_fails=3 fail_timeout=30s;
                    }
                    
                    # 访问30080代理到后端节点
                    server {
                       listen 30080; 
                       proxy_pass k8s-ingress-controller;
                    }
                }
                http {
                    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                                      '$status $body_bytes_sent "$http_referer" '
                                      '"$http_user_agent" "$http_x_forwarded_for"';
                    access_log  /var/log/nginx/access.log  main;
                    sendfile            on;
                    tcp_nopush          on;
                    tcp_nodelay         on;
                    keepalive_timeout   65;
                    types_hash_max_size 2048;
                    include             /etc/nginx/mime.types;
                    default_type        application/octet-stream;
                }
                

                检查配置 && 启动并加入开机自启动

                nginx -t 
                systemctl enable nginx --now
                systemctl status nginx
                

                3、修改Keepalived Master节点配置文件(Keepalived Master操作,我这里将16.32.15.201定义为主)

                mv /etc/keepalived/keepalived.conf{,.$(date +%F)}
                vim /etc/keepalived/keepalived.conf
                vrrp_script check_nginx {
                    script "/etc/keepalived/check_nginx.sh"
                }
                vrrp_instance VI_1 { 
                    state MASTER 
                    interface ens33  # 网卡名称
                    virtual_router_id 51 
                    priority 100 
                    advert_int 1   
                    authentication { 
                        auth_type PASS      
                        auth_pass 1111 
                    }  
                    # 虚拟IP
                    virtual_ipaddress { 
                        16.32.15.100/24
                    } 
                    track_script {
                        check_nginx
                    } 
                }
                

                添加判断Nginx是否运行脚本

                vim /etc/keepalived/check_nginx.sh
                #!/bin/bash
                nc -z localhost 30080
                if [[ $? -ne 0 ]];then
                    systemctl stop keepalived.service
                fi
                chmod +x /etc/keepalived/check_nginx.sh
                

                启动主节点keepalived

                systemctl enable keepalived --now
                

                4、修改Keepalived Backup节点配置文件(Keepalived Master操作,我这里将16.32.15.202定义为备)

                mv /etc/keepalived/keepalived.conf{,.$(date +%F)}
                vim /etc/keepalived/keepalived.conf
                vrrp_script check_nginx {
                    script "/etc/keepalived/check_nginx.sh"
                }
                vrrp_instance VI_1 { 
                    state BACKUP 
                    interface ens33  # 网卡名称
                    virtual_router_id 51 
                    priority 90
                    advert_int 1   
                    authentication { 
                        auth_type PASS      
                        auth_pass 1111 
                    }  
                    # 虚拟IP
                    virtual_ipaddress { 
                        16.32.15.100/24
                    } 
                    track_script {
                        check_nginx
                    } 
                }
                

                添加判断Nginx是否运行脚本

                vim /etc/keepalived/check_nginx.sh
                #!/bin/bash
                nc -z localhost 30080
                if [[ $? -ne 0 ]];then
                    systemctl stop keepalived.service
                fi
                chmod +x /etc/keepalived/check_nginx.sh
                

                启动备节点keepalived

                systemctl enable keepalived --now
                

                3、测试主备切换

                1、在主机停止nginx服务

                systemctl stop nginx
                

                2、在备机,查看VIP是否漂移过去

                ip a|grep 100
                

                如果漂移过去表示无问题,如下图:

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-TUQuELl8-1690102684604)(D:\MD归档文档\IMG\image-20230723143033758.png)],第5张

                3、在主机启动,VIP会自动表漂移到主机

                systemctl start nginx keepalived
                ip a|grep 100
                

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ghq73yAv-1690102684604)(D:\MD归档文档\IMG\image-20230723143203306.png)],第6张

                三、实践:创建Ingress规则进行七层转发

                Ingress规则官方参考文档

                1、基于HTTP七层代理转发后端Pod

                1、创建后端Pod、Server资源

                cat ingress-demo.yaml
                ---
                apiVersion: v1
                kind: Service
                metadata:
                  name: ingress-tomcat-service
                  namespace: default
                spec:
                  selector:
                    app: tomcat
                  ports:
                  - name: http
                    targetPort: 8080
                    port: 8080
                  - name: ajp
                    targetPort: 8009
                    port: 8009
                ---
                apiVersion: apps/v1
                kind: Deployment
                metadata:
                  name: ingress-tomcat-deployment
                  namespace: default
                spec:
                  replicas: 2
                  selector:
                    matchLabels:
                      app: tomcat
                  template:
                    metadata:
                      labels:
                        app: tomcat
                    spec:
                      containers:
                      - name: tomcat
                        image: tomcat:8.5.34-jre8-alpine 
                        imagePullPolicy: IfNotPresent  
                        ports:
                        - name: http
                          containerPort: 8080
                          name: ajp
                          containerPort: 8009
                

                执行YAML文件:

                kubectl apply -f ingress-demo.yaml
                

                查看创建的Pod、Service

                kubectl get pods,svc
                

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ewDW110C-1690102684604)(D:\MD归档文档\IMG\image-20230723150303523.png)],第7张

                2、创建Ingress转发规则

                cat ingress-tomcat.yaml
                apiVersion: networking.k8s.io/v1
                kind: Ingress
                metadata:
                  name: ingress-tomcat
                  namespace: default
                spec: 
                  ingressClassName: nginx     # 指定ingress类名称,这里是Nginx
                  rules:
                  - host: tomcat.ingress.com  # 客户端访问的域名
                    http:
                      paths:
                      - backend:
                          service:
                            name: ingress-tomcat-service  # 转发到SVC名称
                            port:
                              number: 8080                # 转发到SVC端口
                        path: /                           # 转发到/
                        pathType: Prefix
                

                执行YAML

                kubectl apply -f ingress-tomcat.yaml
                

                3、添加域名解析:

                打开 C:\Windows\System32\drivers\etc\hosts 文件,添加解析,如下图:

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-904hIlOE-1690102684605)(D:\MD归档文档\IMG\image-20230723151540821.png)],第8张

                浏览器访问tomcat.ingress.com:30080进行测试

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ZIvwYI4w-1690102684605)(D:\MD归档文档\IMG\image-20230723151720613.png)],第9张

                2、基于HTTPS七层代理转发后端

                基于上面 HTTP七层代理转发的 Pod、Service做实验,不在创建新的资源。

                1、创建证书

                生成一个私钥

                openssl genrsa -out tls.key 2048
                

                基于私钥生成根证书,并签发qinzt.ingress.com 域名

                openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=Beijing/L=Beijing/O=DevOps/CN=qinzt.ingress.com
                

                2、创建secret,对证书进行加密

                kubectl create secret tls ingress-tomcat-secret --cert=tls.crt --key=tls.key
                

                查看secret

                kubectl describe secret ingress-tomcat-secret
                

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-tY1SB7IC-1690102684605)(D:\MD归档文档\IMG\image-20230723155504076.png)],第10张

                3、创建ingress规则

                cat ingress-tomcat-tls.yaml
                apiVersion: networking.k8s.io/v1
                kind: Ingress
                metadata:
                  name: ingress-tomcat-tls
                  namespace: default
                spec:
                  ingressClassName: nginx
                  tls:
                  - hosts:
                    -  qinzt.ingress.com
                    secretName: ingress-tomcat-secret   # secret名称
                  rules:
                  - host: qinzt.ingress.com
                    http:
                      paths:
                      - path: /
                        pathType:  Prefix
                        backend:
                         service:
                           name: tomcat
                           port:
                            number: 8080
                

                执行YAML文件:

                kubectl apply -f ingress-tomcat-tls.yaml
                

                4、添加域名解析:

                打开 C:\Windows\System32\drivers\etc\hosts 文件,添加解析,如下图:

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-QTgim04r-1690102684605)(D:\MD归档文档\IMG\image-20230723160113701.png)],第11张

                5、浏览器访问域名测试

                由于证书是自签名,所有浏览器会提示不安全,点击确认继续访问即可

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-wCHoPi0f-1690102684606)(D:\MD归档文档\IMG\image-20230723160301173.png)],第12张

                【Kubernetes资源篇】ingress-nginx最佳实践详解,[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-JSoIXDiW-1690102684606)(D:\MD归档文档\IMG\image-20230723161554949.png)],第13张