kubernetes1.5.2集群部署过程--安全模式

使用https安全模式部署kubernetes集群，能保证集群通讯安全、有效限制非授权用户访问。但部署比非安全模式复杂的多。

本文为etcd、kubernetes集群中各个组件配置证书认证，所有组件通讯之间使用https通讯。

运行环境

宿主机：CentOS7 7.3.1611
关闭selinux
etcd 3.1.9
flunnel 0.7.1
docker 1.12.6
kubernetes 1.5.2

安装软件

yum install etcd kubernetes kubernetes-client kubernetes-master kubernetes-node flannel docker docker-devel docker-client docker-common -y

证书部署

cfssl

CFSSL是开源的PKI工具箱，可以创建一个轻松获取和操作证书的内部CA。该工具具有运行一个CA所需的全部功能。

运行CA需要一个CA证书和相应的私钥。私钥是极其敏感的数据，任何知道私钥的人都可以充当CA颁发证书，私钥的保护至关重要。

安装cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod a+x cfssl*
mv cfssl-certinfo_linux-amd64 cfssl-certinfo
mv cfssl_linux-amd64 cfssl
mv cfssljson_linux-amd64 cfssljson

签发证书

创建CA证书

创建 CA 配置文件

mkdir /root/ssl
cd /root/ssl 
cat << EOF > ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

字段说明

ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示client可以用该 CA 对server提供的证书进行验证；
client auth：表示server可以用该CA对client提供的证书进行验证；

创建 CA 证书签名请求

cat << EOF > ca-csr.json
{
  "CN": "lykops.net",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "GD",
      "L": "SZ",
      "O": "lykops.net",
      "OU": "lykops.net"
    }
  ]
}
EOF

生成 CA 证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

签发kube-master证书

cat << EOF > kube-master-csr.json
{
    "CN": "kube-master",
    "hosts": [
      "127.0.0.1",
      "192.168.20.128",
      "192.168.20.131",
      "192.168.20.132",
      "172.16.0.1",
      "172.17.0.1",
      "localhost",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.lykops.net",
      "kubernetes.kube-system",
      "kubernetes.kube-system.svc",
      "kubernetes.kube-system.svc.lykops.net"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "SZ",
            "L": "GD",
            "O": "kube-master",
            "OU": "lykops.net"
        }
    ]
}
EOF

如果hosts字段不为空则需要指定授权使用该证书的IP或域名列表。哪些主机需要访问，在hosts中指定。

生成证书和私钥

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-master-csr.json | cfssljson -bare kube-master

或者直接在命令行上指定相关参数：

echo '{"CN":"kubernetes","hosts":[""],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes -hostname="127.0.0.1,kubernetes,kubernetes.default" - | cfssljson -bare kubernetes

签发kubelet证书

cat << EOF > kubelet-csr.json
{
    "CN": "kubelet",
    "hosts": [
      "127.0.0.1",
      "192.168.20.128",
      "192.168.20.131",
      "192.168.20.132",
      "172.16.0.1",
      "172.17.0.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.lykops.net",
      "kubernetes.kube-system",
      "kubernetes.kube-system.svc",
      "kubernetes.kube-system.svc.lykops.net"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "SZ",
            "L": "GD",
            "O": "kubelet",
            "OU": "lykops.net"
        }
    ]
}
EOF

生成证书和私钥

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubelet-csr.json | cfssljson -bare kubelet

签发etcd证书

客户端连接证书

cat << EOF > etcd-client-csr.json
{
    "CN": "etcd-client",
    "hosts": [
      "127.0.0.1",
      "192.168.20.128"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "SZ",
            "L": "GD",
            "O": "etcd-client",
            "OU": "lykops.net"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-client-csr.json | cfssljson -bare etcd-client

集群连接证书

cat << EOF > etcd-member-csr.json
{
    "CN": "etcd-member",
    "hosts": [
      "127.0.0.1",
      "192.168.20.128"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "SZ",
            "L": "GD",
            "O": "etcd-member",
            "OU": "etcd"
        }
    ]
}
EOF

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-member-csr.json | cfssljson -bare etcd-member

校验证书

以kube-master证书为例

使用Opsnssl命令

openssl x509  -noout -text -in  kubernetes.pem
...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=Kubernetes
        Validity
            Not Before: Apr  5 05:36:00 2017 GMT
            Not After : Apr  5 05:36:00 2018 GMT
        Subject: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=System, CN=kubernetes
...

            X509v3 Subject Alternative Name:
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:172.20.0.112, IP Address:172.20.0.113, IP Address:172.20.0.114, IP Address:172.20.0.115, IP Address:10.254.0.1
...

确认Issuer字段的内容和ca-csr.json一致； 确认Subject字段的内容和kubernetes-csr.json一致； 确认X509v3 Subject Alternative Name字段的内容和kubernetes-csr.json一致； 确认X509v3 Key Usage、Extended Key Usage字段的内容和ca-config.json中 kubernetesprofile一致；

使用Cfssl-Certinfo命令

cfssl-certinfo -cert kubernetes.pem
...
{
  "subject": {
    "common_name": "kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "k8s",
      "System",
      "kubernetes"
    ]
  },
  "issuer": {
    "common_name": "Kubernetes",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "BeiJing",
    "province": "BeiJing",
    "names": [
      "CN",
      "BeiJing",
      "BeiJing",
      "k8s",
      "System",
      "Kubernetes"
    ]
  },
  "serial_number": "174360492872423263473151971632292895707129022309",
  "sans": [
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local",
    "127.0.0.1",
    "10.64.3.7",
    "10.254.0.1"
  ],
  "not_before": "2017-04-05T05:36:00Z",
  "not_after": "2018-04-05T05:36:00Z",
  "sigalg": "SHA256WithRSA",
...

下发证书

把etcd、ca全部拷贝到etcd服务器下的/etc/ssl/etcd，设置权限：chown etcd:etcd /etc/ssl/etcd/*

把kube-master和etcd-client、ca全部拷贝到master服务器下的/etc/ssl/kube下，设置权限：chown kube:kube /etc/ssl/kube/

把kubelet、ca、etcd-client全部拷贝到node服务器上的/etc/ssl/kube下，设置权限：chown kube:kube /etc/ssl/kube/

部署etcd

cat /etc/etcd/etcd.conf
# [member]
ETCD_NAME=kube-master
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="https://192.168.20.128:2379,http://localhost:2379,http://localhost:4001"

#[cluster]
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.20.128:2379"

#[security]
ETCD_CERT_FILE="/etc/ssl/etcd/etcd-client.pem"
ETCD_KEY_FILE="/etc/ssl/etcd/etcd-client-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/ssl/etcd/ca.pem"

启动服务service etcd start

flanneld网络

配置flanneld服务

cat /etc/sysconfig/flanneld 
FLANNEL_ETCD_ENDPOINTS="https://192.168.20.128:2379 --etcd-cafile=/etc/ssl/kube/ca.pem --etcd-certfile=/etc/ssl/kube/etcd-client.pem --etcd-keyfile=/etc/ssl/kube/etcd-client-key.pem"
FLANNEL_ETCD_PREFIX="/coreos.com/network"
#FLANNEL_OPTIONS=""

启动flannel服务

创建flannel网络（在etcd服务器上执行）

etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd-client.pem --key-file=/etc/ssl/etcd/etcd-client-key.pem mk /coreos.com/network/config '{"Network":"172.16.0.0/16"}'
etcdctl --ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/etcd-client.pem --key-file=/etc/ssl/etcd/etcd-client-key.pem get /coreos.com/network/config

kube-master

API Server、controller-manager、scheduler三个服务部署在同一台主机上，所以无需使用https通讯，故使用普通的http方式进行通讯。

controllermanager-config

该文件为kubernetes集群中的组件（比如controllermanager等）、addons(比如dashboard等)提供集群组件之间通讯的安全验证配置文件。

其中下面的password、username为访问Server API的认证用户和密码，保存在kube-master服务器上，路径请见API Server配置文件中的--basic-auth-file

cat << EOF > /etc/kubernetes/kube-controllermanager-config
apiVersion: v1
kind: Config
users:
- name: controllermanager
  user:
    client-certificate: /etc/ssl/kube/kube-master.pem
    client-key: /etc/ssl/kube/kube-master-key.pem 
    password: 1qaz2wsx
    username: lykops
clusters:
- name: local
  cluster:
    certificate-authority: /etc/ssl/kube/ca.pem 
    server: https://192.168.20.128:6443
contexts:
- context:
    cluster: local
    user: controllermanager
  name: my-context
current-context: my-context
EOF

apiserver服务

cat /etc/kubernetes/apiserver 
###
# kubernetes system config
# The following values are used to configure the kube-apiserver

# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1 --basic-auth-file=/etc/kubernetes/useraccount.csv"

# The port on the local server to listen on.
KUBE_API_PORT="--insecure-port=8080 --secure-port=6443"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.20.128:2379 --etcd-cafile=/etc/ssl/kube/ca.pem --etcd-certfile=/etc/ssl/kube/etcd-client.pem --etcd-keyfile=/etc/ssl/kube/etcd-client-key.pem"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=172.17.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

# Add your own!
KUBE_API_ARGS="--log-dir=/var/log/kubernetes --client-ca-file=/etc/ssl/kube/ca.pem --tls-private-key-file=/etc/ssl/kube/kube-master-key.pem --tls-cert-file=/etc/ssl/kube/kube-master.pem "

--insecure-bind-address=127.0.0.1表示http端口开放在localhost上

--basic-auth-file=/etc/kubernetes/useraccount.csv登陆账号和密码，必须要配置，否则在后面会出现很多认证失败导致无法通讯的问题。

使用https访问API Server有两种方式：

1、不对称方式：CA证书+用户密码

2、对称方式：CA证书+签发的证书和密钥

重启 kube-apiserver 服务：systemctl restart kube-apiserver

config文件

cat /etc/kubernetes/config
###
# kubernetes system config
# kubernetes services, including
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service

KUBE_LOGTOSTDERR="--logtostderr=false --log-dir=/var/log/kubernetes"
KUBE_LOG_LEVEL="--v=2"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=https://192.168.20.128:6443 --kubeconfig=/etc/kubernetes/kube-controllermanager-config"

Controller Manager服务

/etc/kubernetes/controller-manager 
# The following values are used to configure the kubernetes controller-manager

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--service-account-private-key-file=/etc/ssl/kube/kube-master-key.pem  --root-ca-file=/etc/ssl/kube/ca.pem --master=http://localhost:8080"

scheduler服务

cat /etc/kubernetes/scheduler ### # kubernetes scheduler config KUBESCHEDULERARGS="--master=http://localhost:8080"

proxy服务

cat /etc/kubernetes/proxy 
# kubernetes proxy config
KUBE_PROXY_ARGS="--master=http://localhost:8080"

如果日志报：

kube-controller-manager: E0830 17:08:37.826561    1557 controllermanager.go:558] Failed to start certificate controller: open /etc/kubernetes/ca/ca.pem: no such file or directory

请执行

mkdir /etc/kubernetes/ca/
cp -rpf /etc/ssl/kube/ca.pem /etc/kubernetes/ca/

node

kubelet-config

cat << EOF > /etc/kubernetes/kubelet-config
apiVersion: v1
kind: Config
users:
- name: kubelet
  user:
    client-certificate: /etc/ssl/kube/kubelet.pem
    client-key: /etc/ssl/kube/kubelet-key.pem
    password: 1qaz2wsx
    username: lykops
clusters:
- name: local
  cluster:
  certificate-authority: /etc/ssl/kube/ca.pem
  server: https://192.168.20.128:6443
contexts:
- context:
    cluster: local
    user: kubelet
  name: kubelet-context
current-context: kubelet-context
EOF

kubelet服务

cat /etc/kubernetes/kubelet
###
# kubernetes kubelet (minion) config
# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=kube-node1"

# location of the api-server
KUBELET_API_SERVER="--api-servers=https://192.168.20.128:6443 --client-ca-file=/etc/ssl/kube/ca.pem --tls-private-key-file=/etc/ssl/kube/kubelet-key.pem --tls-cert-file=/etc/ssl/kube/kubelet.pem --kubeconfig=/etc/kubernetes/kubelet-config"

# pod infrastructure container
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_ARGS="--cluster-domain=lykops.net --cluster_dns=172.17.114.114"

config文件

cat /etc/kubernetes/config 
###
# kubernetes system config
# The following values are used to configure various aspects of all
# kubernetes services, including
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=false --log-dir=/var/log/kubernetes"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=2"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=https://192.168.20.128:6443 --kubeconfig=/etc/kubernetes/kubelet-config"

proxy服务为默认