Kubernetes 证书延期

一、概述

kubeadm 是 kubernetes 提供的一个初始化集群的工具，使用起来非常方便，但是它创建的 apiserver、controller-manager 等证书默认只有一年的有效期，同时 kubelet 证书也只有一年有效期，一年之后 kubernetes 将停止服务

Kubernetes 集群根证书：
/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
......

kubeadm 默认生成的ca证书有效期是10年，其他证书（如etcd证书，apiserver证书）有效期均为1年。

二、准备工作

[root@k8s-master etcd]#  tree /etc/kubernetes/
/etc/kubernetes/
├── admin.conf
├── controller-manager.conf
├── kubelet.conf
├── manifests
│   ├── etcd.yaml
│   ├── kube-apiserver.yaml
│   ├── kube-controller-manager.yaml
│   └── kube-scheduler.yaml
├── pki
│   ├── apiserver.crt
│   ├── apiserver-etcd-client.crt
│   ├── apiserver-etcd-client.key
│   ├── apiserver.key
│   ├── apiserver-kubelet-client.crt
│   ├── apiserver-kubelet-client.key
│   ├── ca.crt
│   ├── ca.key
│   ├── etcd
│   │   ├── ca.crt
│   │   ├── ca.key
│   │   ├── healthcheck-client.crt
│   │   ├── healthcheck-client.key
│   │   ├── peer.crt
│   │   ├── peer.key
│   │   ├── server.crt
│   │   └── server.key
│   ├── front-proxy-ca.crt
│   ├── front-proxy-ca.key
│   ├── front-proxy-client.crt
│   ├── front-proxy-client.key
│   ├── sa.key
│   └── sa.pub
└── scheduler.conf

3 directories, 30 files

查看证书:

[root@k8s-master]# cd /etc/kubernetes/pki

[root@k8s-master pki]#  openssl x509 -in front-proxy-client.crt   -noout -text  |grep Not
            Not Before: Jul 29 12:07:53 2020 GMT
            Not After : Jul 29 12:07:54 2021 GMT

[root@k8s-master pki]#  openssl x509 -in apiserver.crt   -noout -text  |grep Not
            Not Before: Jul 29 12:07:52 2020 GMT
            Not After : Jul 29 12:07:53 2021 GMT

[root@k8s-master pki]#  openssl x509 -in front-proxy-client.crt   -noout -text  |grep Not
            Not Before: Aug  1 08:36:22 2019 GMT
            Not After : Jul 31 10:47:04 2021 GMT

[root@k8s-master pki]#  openssl x509 -in apiserver.crt   -noout -text  |grep Not
            Not Before: Aug  1 08:36:24 2019 GMT
            Not After : Jul 31 10:47:04 2021 GMT

............


[root@k8s-master kubernetes]#  find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep Not'
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/server.crt|grep Not
            Not Before: Aug  1 08:36:23 2019 GMT
            Not After : Jul 31 10:47:03 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt|grep Not
            Not Before: Aug  1 08:36:23 2019 GMT
            Not After : Jul 31 10:47:03 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/peer.crt|grep Not
            Not Before: Aug  1 08:36:23 2019 GMT
            Not After : Jul 31 10:47:03 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt|grep Not
            Not Before: Aug  1 08:36:23 2019 GMT
            Not After : Jul 31 10:47:04 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-kubelet-client.crt|grep Not
            Not Before: Aug  1 08:36:24 2019 GMT
            Not After : Jul 31 10:47:04 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver.crt|grep Not
            Not Before: Aug  1 08:36:24 2019 GMT
            Not After : Jul 31 10:47:04 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/front-proxy-client.crt|grep Not
            Not Before: Aug  1 08:36:22 2019 GMT
            Not After : Jul 31 10:47:04 2021 GMT


[root@k8s-master kubernetes]#  find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/server.crt|grep After
            Not After : Jul 31 10:47:03 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt|grep After
            Not After : Jul 31 10:47:03 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/peer.crt|grep After
            Not After : Jul 31 10:47:03 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt|grep After
            Not After : Jul 31 10:47:04 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-kubelet-client.crt|grep After
            Not After : Jul 31 10:47:04 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver.crt|grep After
            Not After : Jul 31 10:47:04 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/front-proxy-client.crt|grep After
            Not After : Jul 31 10:47:04 2021 GMT

[root@k8s-master pki]#  ls -l
总用量 56
-rw-r--r-- 1 root root 1224 7月  31 18:47 apiserver.crt
-rw-r--r-- 1 root root 1090 7月  31 18:47 apiserver-etcd-client.crt
-rw------- 1 root root 1675 7月  31 18:47 apiserver-etcd-client.key
-rw------- 1 root root 1675 7月  31 18:47 apiserver.key
-rw-r--r-- 1 root root 1099 7月  31 18:47 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 7月  31 18:47 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1025 8月   1 2019 ca.crt
-rw------- 1 root root 1675 8月   1 2019 ca.key
drwxr-xr-x 2 root root  162 7月  31 18:47 etcd
-rw-r--r-- 1 root root 1038 8月   1 2019 front-proxy-ca.crt
-rw------- 1 root root 1679 8月   1 2019 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 7月  31 18:47 front-proxy-client.crt
-rw------- 1 root root 1675 7月  31 18:47 front-proxy-client.key
-rw------- 1 root root 1679 8月   1 2019 sa.key
-rw------- 1 root root  451 8月   1 2019 sa.pub
[root@k8s-master pki]#  openssl x509 -in ca.crt   -noout -text  |grep Not
            Not Before: Aug  1 08:36:24 2019 GMT
            Not After : Jul 29 08:36:24 2029 GMT

[root@k8s-master pki]#  cd etcd/
[root@k8s-master etcd]#  ls -l
总用量 32
-rw-r--r-- 1 root root 1017 8月   1 2019 ca.crt
-rw------- 1 root root 1679 8月   1 2019 ca.key
-rw-r--r-- 1 root root 1094 7月  31 18:47 healthcheck-client.crt
-rw------- 1 root root 1675 7月  31 18:47 healthcheck-client.key
-rw-r--r-- 1 root root 1135 7月  31 18:47 peer.crt
-rw------- 1 root root 1679 7月  31 18:47 peer.key
-rw-r--r-- 1 root root 1127 7月  31 18:47 server.crt
-rw------- 1 root root 1675 7月  31 18:47 server.key
[root@k8s-master etcd]#  openssl x509 -in ca.crt   -noout -text  |grep Not
            Not Before: Aug  1 08:36:23 2019 GMT
            Not After : Jul 29 08:36:23 2029 GMT


备份工作（非常重要）:

[root@k8s-master ]# cd /etc/kubernetes
[root@k8s-master kubernetes]# mkdir ./pki_bak
[root@k8s-master kubernetes]# mkdir ./pki_bak/etcd
[root@k8s-master kubernetes]# mkdir ./conf_bak
[root@k8s-master kubernetes]# cp pki/apiserver* ./pki_bak/
[root@k8s-master kubernetes]# cp pki/front-proxy-client.* ./pki_bak/
[root@k8s-master kubernetes]# cp pki/etcd/healthcheck-client.* ./pki_bak/etcd/
[root@k8s-master kubernetes]# cp pki/etcd/peer.* ./pki_bak/etcd/
[root@k8s-master kubernetes]# cp pki/etcd/server.* ./pki_bak/etcd/
[root@k8s-master kubernetes]# cp ./admin.conf ./conf_bak/
[root@k8s-master kubernetes]# cp ./kubelet.conf ./conf_bak/
[root@k8s-master kubernetes]# cp ./controller-manager.conf ./conf_bak/
[root@k8s-master kubernetes]# cp ./scheduler.conf ./conf_bak/


三、证书类别

1、集群根证书：
[root@k8s-master images]#  ll /etc/kubernetes/pki/ca*
-rw-r--r-- 1 root root 1025 8月   1 2019 /etc/kubernetes/pki/ca.crt
-rw------- 1 root root 1675 8月   1 2019 /etc/kubernetes/pki/ca.key


2、由此集群根证书签发的证书有：

1）kube-apiserver 组件持有的服务端证书
[root@k8s-master pki]#  ll /etc/kubernetes/pki/apiserver.*
-rw-r--r-- 1 root root 1224 7月  31 18:47 /etc/kubernetes/pki/apiserver.crt
-rw------- 1 root root 1675 7月  31 18:47 /etc/kubernetes/pki/apiserver.key

2）kubelet 组件持有的客户端证书
[root@k8s-master pki]#  ll /etc/kubernetes/pki/apiserver-kubelet-client.*
-rw-r--r-- 1 root root 1099 7月  31 18:47 /etc/kubernetes/pki/apiserver-kubelet-client.crt
-rw------- 1 root root 1675 7月  31 18:47 /etc/kubernetes/pki/apiserver-kubelet-client.key
 
提示：kubelet的/var/lib/kubelet/config.yaml配置文件中一般不会明确指定服务端证书，而是只指定 ca 根证书, 让 kubelet 根据本地主机信息自动生成服务端证书并保存到配置的 cert-dir文件夹中。

[root@k8s-master kubelet]#  pwd
/var/lib/kubelet
[root@k8s-master kubelet]#  cat config.yaml
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
configMapAndSecretChangeDetectionStrategy: Watch
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuCFSQuotaPeriod: 100ms
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kind: KubeletConfiguration
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeLeaseDurationSeconds: 40
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
port: 10250
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

3）汇聚层证书
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/front-proxy-ca.*
-rw-r--r-- 1 root root 1038 8月   1 2019 /etc/kubernetes/pki/front-proxy-ca.crt
-rw------- 1 root root 1679 8月   1 2019 /etc/kubernetes/pki/front-proxy-ca.key

由此汇聚层根证书签发的证书有：
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/front-proxy-client.*
-rw-r--r-- 1 root root 1058 7月  31 18:47 /etc/kubernetes/pki/front-proxy-client.crt
-rw------- 1 root root 1675 7月  31 18:47 /etc/kubernetes/pki/front-proxy-client.key

3、etcd集群根证书:
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/etcd/ca.*
-rw-r--r-- 1 root root 1017 8月   1 2019 /etc/kubernetes/pki/etcd/ca.crt
-rw------- 1 root root 1679 8月   1 2019 /etc/kubernetes/pki/etcd/ca.key

由此etcd根证书签发的证书有：

etcd server服务端证书:
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/etcd/server.*
-rw-r--r-- 1 root root 1127 7月  31 18:47 /etc/kubernetes/pki/etcd/server.crt
-rw------- 1 root root 1675 7月  31 18:47 /etc/kubernetes/pki/etcd/server.key


etcd 集群中peer节点互相通信使用的客户端证书:
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/etcd/peer.*
-rw-r--r-- 1 root root 1135 7月  31 18:47 /etc/kubernetes/pki/etcd/peer.crt
-rw------- 1 root root 1679 7月  31 18:47 /etc/kubernetes/pki/etcd/peer.key


pod 中定义 Liveness 探针使用的客户端证书:
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/etcd/healthcheck-client.*
-rw-r--r-- 1 root root 1094 7月  31 18:47 /etc/kubernetes/pki/etcd/healthcheck-client.crt
-rw------- 1 root root 1675 7月  31 18:47 /etc/kubernetes/pki/etcd/healthcheck-client.key


配置在 kube-apiserver 中用来与 etcd server 做双向认证的客户端证书:
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/apiserver-etcd-client.*
-rw-r--r-- 1 root root 1090 7月  31 18:47 /etc/kubernetes/pki/apiserver-etcd-client.crt
-rw------- 1 root root 1675 7月  31 18:47 /etc/kubernetes/pki/apiserver-etcd-client.key

4、Serveice Account密钥:
[root@k8s-master kubelet]#  ll /etc/kubernetes/pki/sa.*
-rw------- 1 root root 1679 8月   1 2019 /etc/kubernetes/pki/sa.key
-rw------- 1 root root  451 8月   1 2019 /etc/kubernetes/pki/sa.pub
 
 
Serveice Account密钥对仅提供给 kube-controller-manager 使用. kube-controller-manager 通过 sa.key 对 token 进行签名, master 节点通过公钥 sa.pub 进行签名的验证。

API Server身份验证过程：
API Server的authenticating环节支持多种身份校验方式：client cert、bearer token、static password auth等，这些方式中只要有一种方式通过authenticating（Kubernetes API Server会逐个方式尝试），那么身份校验就会通过。
一旦API Server发现client发起的request使用的是service account token的方式，API Server就会自动采用signed bearer token方式进行身份校验。而request则使用携带的service account token参与验证。该token是API Server在创建service account时用API server启动参数：–service-account-key-file的值签署(sign)生成的。如果–service-account-key-file未传入任何值，那么将默认使用–tls-private-key-file的值，即API Server的私钥（server.key）。
通过authenticating后，API Server将根据Pod username所在的group：system:serviceaccounts和system:serviceaccounts:(NAMESPACE)的权限对其进行authority 和admission control两个环节的处理。在这两个环节中，cluster管理员可以对service account的权限进行细化设置。

kubeadm 创建的集群，kube-proxy、flannel、coreDNS是以 pod 形式运行的，在 pod 中，直接使用 service account 与 kube-apiserver 进行认证，此时就不需要再单独为 kube-proxy 创建证书。


---------------------------------------------kubernetes 1.15 版本 以下方案---------------------------------------------

提示：1.12.1 使用可用

1、查看kubeadm-config配置

提示：不同的master节点使用的kubeadm配置有细微的差异，执行更新证书是，每个master在--config后面使用原来集群创建时，当前master对应的kubeadm配置文件。

[root@k8s-master kubernetes]# kubectl -n kube-system get cm kubeadm-config -oyaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServerExtraArgs:
      authorization-mode: Node,RBAC
    apiVersion: kubeadm.k8s.io/v1alpha3
    auditPolicy:
      logDir: /var/log/kubernetes/audit
      logMaxAge: 2
      path: ""
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controlPlaneEndpoint: ""
    etcd:
      local:
        dataDir: /var/lib/etcd
        image: ""
    imageRepository: k8s.gcr.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.12.1
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    unifiedControlPlaneImage: ""
  ClusterStatus: |
    apiEndpoints:
      k8s-master:
        advertiseAddress: 192.101.10.80
        bindPort: 6443
    apiVersion: kubeadm.k8s.io/v1alpha3
    kind: ClusterStatus
kind: ConfigMap
metadata:
  creationTimestamp: 2019-08-01T08:36:48Z
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "174"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config
  uid: 80959d9d-b437-11e9-8e73-6c92bfa51bf6

2、创建kubeadm-cluster.yaml

# touch kubeadm-cluster.yaml
# vi kubeadm-cluster.yaml

apiServer:
  apiServerExtraArgs:
   authorization-mode: Node,RBAC
apiVersion: kubeadm.k8s.io/v1alpha3
auditPolicy:
  logDir: /var/log/kubernetes/audit
  logMaxAge: 2
  path: ""
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.12.1
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
unifiedControlPlaneImage: ""


3、查看帮助

提示：不同版本的kubeadm对于证书renew的命令有细微的差异，具体情况需要依据已经安装的kubeadm来判断。通过命令行kubeadm alpha --help输出类似如下信息：

[root@k8s-master yaml]# kubeadm alpha --help
Experimental sub-commands not yet fully functional.

Usage:
  kubeadm alpha [command]

Available Commands:
  phase       Invoke subsets of kubeadm functions separately for a manual install.

Flags:
  -h, --help   help for alpha

Global Flags:
      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.
  -v, --v Level         log level for V logs

Use "kubeadm alpha [command] --help" for more information about a command.


[root@k8s-master yaml]# kubeadm alpha phase certs --help
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm alpha phase certs [command]

Aliases:
  certs, certificates

Available Commands:
  all                      Generates all PKI assets necessary to establish the control plane
  apiserver                Generates the certificate for serving the kubernetes API
  apiserver-etcd-client    Generates the client apiserver uses to access etcd
  apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet
  ca                       Generates the self-signed kubernetes CA to provision identities for other kuberenets components
  etcd-ca                  Generates the self-signed CA to provision identities for etcd
  etcd-healthcheck-client  Generates the client certificate for liveness probes to healtcheck etcd
  etcd-peer                Generates the credentials for etcd nodes to communicate with each other
  etcd-server              Generates the certificate for serving etcd
  front-proxy-ca           Generates the self-signed CA to provision identities for front proxy
  front-proxy-client       Generates the client for the front proxy
  renew                    Renews certificates for a Kubernetes cluster
  sa                       Generates a private key for signing service account tokens along with its public key

Flags:
  -h, --help   help for certs

Global Flags:
      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.
  -v, --v Level         log level for V logs

Use "kubeadm alpha phase certs [command] --help" for more information about a command.

[root@k8s-master yaml]# kubeadm alpha phase certs renew --help
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm alpha phase certs renew [flags]
  kubeadm alpha phase certs renew [command]

Available Commands:
  all                      renew all available certificates
  apiserver                Generates the certificate for serving the kubernetes API
  apiserver-etcd-client    Generates the client apiserver uses to access etcd
  apiserver-kubelet-client Generates the Client certificate for the API server to connect to kubelet
  etcd-healthcheck-client  Generates the client certificate for liveness probes to healtcheck etcd
  etcd-peer                Generates the credentials for etcd nodes to communicate with each other
  etcd-server              Generates the certificate for serving etcd
  front-proxy-client       Generates the client for the front proxy

Flags:
  -h, --help   help for renew

Global Flags:
      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.
  -v, --v Level         log level for V logs

Use "kubeadm alpha phase certs renew [command] --help" for more information about a command.


4、重新生成master各个证书

kubeadm alpha phase certs renew  etcd-healthcheck-client --config kubeadm-cluster.yaml
kubeadm alpha phase certs renew  etcd-peer --config kubeadm-cluster.yaml
kubeadm alpha phase certs renew  etcd-server --config kubeadm-cluster.yaml
kubeadm alpha phase certs renew front-proxy-client --config kubeadm-cluster.yaml
kubeadm alpha phase certs renew  apiserver-etcd-client --config kubeadm-cluster.yaml
kubeadm alpha phase certs renew  apiserver-kubelet-client --config kubeadm-cluster.yaml
kubeadm alpha phase certs renew  apiserver --config kubeadm-cluster.yaml


5、验证证书有效期更新

[root@k8s-master images]# find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/server.crt|grep After
            Not After : Aug  1 10:20:09 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/healthcheck-client.crt|grep After
            Not After : Aug  1 10:19:54 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/etcd/peer.crt|grep After
            Not After : Aug  1 10:20:02 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-etcd-client.crt|grep After
            Not After : Aug  1 10:20:25 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver-kubelet-client.crt|grep After
            Not After : Aug  1 10:20:32 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/apiserver.crt|grep After
            Not After : Aug  1 10:20:39 2021 GMT
bash -c openssl x509  -noout -text -in /etc/kubernetes/pki/front-proxy-client.crt|grep After
            Not After : Aug  1 10:20:18 2021 GMT


6、更新各个配置文件

1）查看帮助：

[root@k8s-master manifests]# kubeadm alpha phase kubeconfig --help
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm alpha phase kubeconfig [command]

Available Commands:
  admin              Generates a kubeconfig file for the admin to use and for kubeadm itself
  all                Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  controller-manager Generates a kubeconfig file for the controller manager to use
  kubelet            Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes
  scheduler          Generates a kubeconfig file for the scheduler to use
  user               Outputs a kubeconfig file for an additional user

Flags:
  -h, --help   help for kubeconfig

Global Flags:
      --rootfs string   [EXPERIMENTAL] The path to the 'real' host root filesystem.
  -v, --v Level         log level for V logs

Use "kubeadm alpha phase kubeconfig [command] --help" for more information about a command.


2）备份集群配置文件并重新生成：

[root@k8s-master yaml]# find /etc/kubernetes/ -name '*.conf'|xargs -i mv {}{,bak}

[root@k8s-master yaml]# kubeadm alpha phase kubeconfig all --config kubeadm-cluster.yaml
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
或者分步执行：
kubeadm alpha phase kubeconfig kubelet --config kubeadm-cluster.yaml
kubeadm alpha phase kubeconfig  admin --config kubeadm-cluster.yaml
kubeadm alpha phase kubeconfig  scheduler --config kubeadm-cluster.yaml
kubeadm alpha phase kubeconfig  controller-manager --config kubeadm-cluster.yaml


3）重新配置kubectl权限信息：

mv $HOME/.kube/config $HOME/.kube/config.old
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config


7、更新集群验证

验证kubernetes 集群：运行 kubectl cluster-info 和 kubectl get nodes 符合预期

[root@k8s-master images]# kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
k8s-master   Ready     master    1y        v1.12.1
k8s-node1    Ready     <none>    1y        v1.12.1
k8s-node2    Ready     <none>    1y        v1.12.1

确性kubernetes 系统相关的服务运行正常(核心是kube-apiserver,kube-controller-manager,kube-proxy, kube-flannel)：kubectl get pods -n kube-system

[root@k8s-master images]# kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
coredns-576cbf47c7-fk7j9                1/1       Running   0          1y
coredns-576cbf47c7-p4f4q                1/1       Running   0          1y
etcd-k8s-master                         1/1       Running   2          1y
kube-apiserver-k8s-master               1/1       Running   0          2h
kube-controller-manager-k8s-master      1/1       Running   1          2h
kube-flannel-ds-amd64-f2csl             1/1       Running   0          1y
kube-flannel-ds-amd64-wm2b6             1/1       Running   0          1y
kube-flannel-ds-amd64-wrnnk             1/1       Running   1          1y
kube-proxy-cz5xg                        1/1       Running   0          1y
kube-proxy-fnr96                        1/1       Running   0          1y
kube-proxy-xbrcb                        1/1       Running   0          1y
kube-scheduler-k8s-master               1/1       Running   1          1y
kubernetes-dashboard-77fd78f978-jl98q   1/1       Running   0          218d

检查pod的运行状态：kubectl get pods --all-namespaces

[root@k8s-master images]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
default       bbcx-service-7c4bb5456-bksqb            1/1       Running            0          7h
default       bbcx-vue-5569488679-zd75m               1/1       Running            1          7h
default       cip-data-service-66ffd668dd-2l4wv       1/1       Running            0          17d
default       cip-job-5fb59f9d84-4lrb4                1/1       Running            0          17d
default       consul-0                                1/1       Running            0          193d
...................


---------------------------------------------kubernetes 1.15 版本 以上方案---------------------------------------------

提示：kubernetes 1.16.3 ；1.18.2 使用可用，1.12.1 使用不可用

第一种方案：

1、查看具体过期时间

kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Jul 29, 2021 12:07 UTC   362d            no     
apiserver                  Jul 29, 2021 12:07 UTC   362d            no     
apiserver-etcd-client      Jul 29, 2021 12:07 UTC   362d            no     
apiserver-kubelet-client   Jul 29, 2021 12:07 UTC   362d            no     
controller-manager.conf    Jul 29, 2021 12:07 UTC   362d            no     
etcd-healthcheck-client    Jul 29, 2021 12:07 UTC   362d            no     
etcd-peer                  Jul 29, 2021 12:07 UTC   362d            no     
etcd-server                Jul 29, 2021 12:07 UTC   362d            no     
front-proxy-client         Jul 29, 2021 12:07 UTC   362d            no     
scheduler.conf             Jul 29, 2021 12:07 UTC   362d            no  

2、查看帮助

[root@hadoop009 images]# kubeadm alpha certs renew --help
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm alpha certs renew [flags]
  kubeadm alpha certs renew [command]

Available Commands:
  admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
  all                      Renew all available certificates
  apiserver                Renew the certificate for serving the Kubernetes API
  apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd
  apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
  controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use
  etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd
  etcd-peer                Renew the certificate for etcd nodes to communicate with each other
  etcd-server              Renew the certificate for serving etcd
  front-proxy-client       Renew the certificate for the front proxy client
  scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

Flags:
  -h, --help   help for renew

Global Flags:
      --add-dir-header           If true, adds the file directory to the header
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm alpha certs renew [command] --help" for more information about a command.

提示：由help可知，证书更新可针对单个证书更新


3、更新证书

更新所有证书，对证书进行续期,续期一年：
$ kubeadm alpha  certs renew all

[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

提示：更新操作需要在所有master节点执行

4、验证续期

# ls -l
总用量 56
-rw-r--r-- 1 root root 1220 7月  31 22:48 apiserver.crt
-rw-r--r-- 1 root root 1090 7月  31 22:48 apiserver-etcd-client.crt
-rw------- 1 root root 1679 7月  31 22:48 apiserver-etcd-client.key
-rw------- 1 root root 1675 7月  31 22:48 apiserver.key
-rw-r--r-- 1 root root 1099 7月  31 22:48 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 7月  31 22:48 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1025 7月  29 20:07 ca.crt
-rw------- 1 root root 1679 7月  29 20:07 ca.key
drwxr-xr-x 2 root root  162 7月  29 20:07 etcd
-rw-r--r-- 1 root root 1038 7月  29 20:07 front-proxy-ca.crt
-rw------- 1 root root 1675 7月  29 20:07 front-proxy-ca.key
-rw-r--r-- 1 root root 1058 7月  31 22:48 front-proxy-client.crt
-rw------- 1 root root 1679 7月  31 22:48 front-proxy-client.key
-rw------- 1 root root 1675 7月  29 20:07 sa.key
-rw------- 1 root root  451 7月  29 20:07 sa.pub

# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 01, 2021 07:38 UTC   364d                                    no     
apiserver                  Aug 01, 2021 07:38 UTC   364d            ca                      no     
apiserver-etcd-client      Aug 01, 2021 07:38 UTC   364d            etcd-ca                 no     
apiserver-kubelet-client   Aug 01, 2021 07:38 UTC   364d            ca                      no     
controller-manager.conf    Aug 01, 2021 07:38 UTC   364d                                    no     
etcd-healthcheck-client    Aug 01, 2021 07:38 UTC   364d            etcd-ca                 no     
etcd-peer                  Aug 01, 2021 07:38 UTC   364d            etcd-ca                 no     
etcd-server                Aug 01, 2021 07:38 UTC   364d            etcd-ca                 no     
front-proxy-client         Aug 01, 2021 07:38 UTC   364d            front-proxy-ca          no     
scheduler.conf             Aug 01, 2021 07:38 UTC   364d                                    no     

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 27, 2030 13:01 UTC   9y              no     
etcd-ca                 Jul 27, 2030 13:01 UTC   9y              no     
front-proxy-ca          Jul 27, 2030 13:01 UTC   9y              no   

[root@hadoop010 etcd]# kubectl -n kube-system get cm kubeadm-config -oyaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta2
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controlPlaneEndpoint: 192.101.11.162:6443
    controllerManager: {}
    dns:
      type: CoreDNS
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: k8s.gcr.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.18.2
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
  ClusterStatus: |
    apiEndpoints:
      hadoop010:
        advertiseAddress: 192.101.11.162
        bindPort: 6443
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterStatus
kind: ConfigMap
metadata:
  creationTimestamp: "2020-07-29T13:02:16Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:ClusterConfiguration: {}
        f:ClusterStatus: {}
    manager: kubeadm
    operation: Update
    time: "2020-07-29T13:02:16Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "157"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config
  uid: 2e049082-fa64-4e2e-ad73-af9fc94a051e


5、启用证书

在每台Master上执行重启kube-apiserver、kube-controller、kube-scheduler、etcd这4个容器，以便使证书生效。

# docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart
或者
# docker ps | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | awk '{print $1}' | awk '{print "docker","restart",$1}' | bash
 
提示：启用操作需要在所有master节点执行。

6、更新.kube下的配置文件

$ mv $HOME/.kube/config $HOME/.kube/config.old
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodes


7、更新集群验证

验证kubernetes 集群：运行 kubectl cluster-info 和 kubectl get nodes 符合预期。
[root@k8s-master images]# kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
k8s-master   Ready     master    1y        v1.12.1
k8s-node1    Ready     <none>    1y        v1.12.1
k8s-node2    Ready     <none>    1y        v1.12.1

确性kubernetes 系统相关的服务运行正常(核心是kube-apiserver,kube-controller-manager,kube-proxy, kube-flannel)：kubectl get pods -n kube-system
[root@k8s-master images]# kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
coredns-576cbf47c7-fk7j9                1/1       Running   0          1y
coredns-576cbf47c7-p4f4q                1/1       Running   0          1y
etcd-k8s-master                         1/1       Running   2          1y
kube-apiserver-k8s-master               1/1       Running   0          2h
kube-controller-manager-k8s-master      1/1       Running   1          2h
kube-flannel-ds-amd64-f2csl             1/1       Running   0          1y
kube-flannel-ds-amd64-wm2b6             1/1       Running   0          1y
kube-flannel-ds-amd64-wrnnk             1/1       Running   1          1y
kube-proxy-cz5xg                        1/1       Running   0          1y
kube-proxy-fnr96                        1/1       Running   0          1y
kube-proxy-xbrcb                        1/1       Running   0          1y
kube-scheduler-k8s-master               1/1       Running   1          1y
kubernetes-dashboard-77fd78f978-jl98q   1/1       Running   0          218d

检查pod的运行状态：kubectl get pods --all-namespaces
[root@k8s-master images]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
default       bbcx-service-7c4bb5456-bksqb            1/1       Running            0          7h
default       bbcx-vue-5569488679-zd75m               1/1       Running            1          7h
default       cip-data-service-66ffd668dd-2l4wv       1/1       Running            0          17d
default       cip-job-5fb59f9d84-4lrb4                1/1       Running            0          17d
default       consul-0                                1/1       Running            0          193d
...................


第二种方案：

1、备份导出kubeadm集群配置

# kubeadm config view > kubeadm-cluster.yaml

apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.16.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

2、查看具体过期时间

kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Jul 29, 2021 12:07 UTC   362d            no     
apiserver                  Jul 29, 2021 12:07 UTC   362d            no     
apiserver-etcd-client      Jul 29, 2021 12:07 UTC   362d            no     
apiserver-kubelet-client   Jul 29, 2021 12:07 UTC   362d            no     
controller-manager.conf    Jul 29, 2021 12:07 UTC   362d            no     
etcd-healthcheck-client    Jul 29, 2021 12:07 UTC   362d            no     
etcd-peer                  Jul 29, 2021 12:07 UTC   362d            no     
etcd-server                Jul 29, 2021 12:07 UTC   362d            no     
front-proxy-client         Jul 29, 2021 12:07 UTC   362d            no     
scheduler.conf             Jul 29, 2021 12:07 UTC   362d            no  

3、查看帮助

[root@hadoop009 images]# kubeadm alpha certs renew --help
This command is not meant to be run on its own. See list of available subcommands.

Usage:
  kubeadm alpha certs renew [flags]
  kubeadm alpha certs renew [command]

Available Commands:
  admin.conf               Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
  all                      Renew all available certificates
  apiserver                Renew the certificate for serving the Kubernetes API
  apiserver-etcd-client    Renew the certificate the apiserver uses to access etcd
  apiserver-kubelet-client Renew the certificate for the API server to connect to kubelet
  controller-manager.conf  Renew the certificate embedded in the kubeconfig file for the controller manager to use
  etcd-healthcheck-client  Renew the certificate for liveness probes to healthcheck etcd
  etcd-peer                Renew the certificate for etcd nodes to communicate with each other
  etcd-server              Renew the certificate for serving etcd
  front-proxy-client       Renew the certificate for the front proxy client
  scheduler.conf           Renew the certificate embedded in the kubeconfig file for the scheduler manager to use

Flags:
  -h, --help   help for renew

Global Flags:
      --add-dir-header           If true, adds the file directory to the header
      --log-file string          If non-empty, use this log file
      --log-file-max-size uint   Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --rootfs string            [EXPERIMENTAL] The path to the 'real' host root filesystem.
      --skip-headers             If true, avoid header prefixes in the log messages
      --skip-log-headers         If true, avoid headers when opening log files
  -v, --v Level                  number for the log level verbosity

Use "kubeadm alpha certs renew [command] --help" for more information about a command.

提示：由help可知，证书更新可针对单个证书更新

4、更新证书

# kubeadm alpha certs renew all --config=kubeadm-cluster.yaml   #更新所有证书
提示：在保存kubeadm-cluster.yaml文件的目录下执行
提示：更新操作需要在所有master节点执行

5、确认验证

# kubeadm alpha certs check-expiration 


6、启用证书

在每台Master上执行重启kube-apiserver、kube-controller、kube-scheduler、etcd这4个容器，以便使证书生效。

# docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart
或者
# docker ps | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | awk '{print $1}' | awk '{print "docker","restart",$1}' | bash
 
提示：启用操作需要在所有master节点执行。

7、更新.kube下的配置文件

$ mv $HOME/.kube/config $HOME/.kube/config.old
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl get nodes


8、更新集群验证

验证kubernetes 集群：运行 kubectl cluster-info 和 kubectl get nodes 符合预期。
[root@k8s-master images]# kubectl get nodes
NAME         STATUS    ROLES     AGE       VERSION
k8s-master   Ready     master    1y        v1.12.1
k8s-node1    Ready     <none>    1y        v1.12.1
k8s-node2    Ready     <none>    1y        v1.12.1

确性kubernetes 系统相关的服务运行正常(核心是kube-apiserver,kube-controller-manager,kube-proxy, kube-flannel)：kubectl get pods -n kube-system
[root@k8s-master images]# kubectl get pods -n kube-system
NAME                                    READY     STATUS    RESTARTS   AGE
coredns-576cbf47c7-fk7j9                1/1       Running   0          1y
coredns-576cbf47c7-p4f4q                1/1       Running   0          1y
etcd-k8s-master                         1/1       Running   2          1y
kube-apiserver-k8s-master               1/1       Running   0          2h
kube-controller-manager-k8s-master      1/1       Running   1          2h
kube-flannel-ds-amd64-f2csl             1/1       Running   0          1y
kube-flannel-ds-amd64-wm2b6             1/1       Running   0          1y
kube-flannel-ds-amd64-wrnnk             1/1       Running   1          1y
kube-proxy-cz5xg                        1/1       Running   0          1y
kube-proxy-fnr96                        1/1       Running   0          1y
kube-proxy-xbrcb                        1/1       Running   0          1y
kube-scheduler-k8s-master               1/1       Running   1          1y
kubernetes-dashboard-77fd78f978-jl98q   1/1       Running   0          218d

检查pod的运行状态：kubectl get pods --all-namespaces
[root@k8s-master images]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
default       bbcx-service-7c4bb5456-bksqb            1/1       Running            0          7h
default       bbcx-vue-5569488679-zd75m               1/1       Running            1          7h
default       cip-data-service-66ffd668dd-2l4wv       1/1       Running            0          17d
default       cip-job-5fb59f9d84-4lrb4                1/1       Running            0          17d
default       consul-0                                1/1       Running            0          193d
...................