使用kubeadm工具部署kubernetes集群

发表于 | 分类于 |

本文讲述使用kubeadm工具安装部署kubernetes集群，本文假设你的机器一些准备工作已做完，例如hosts映射配置、机器间免密登录等。

1. 安装前准备 (以下操作需要在所有节点执行)

环境需求

系统：CentOS7.4 +

硬件需求：CPU >= 2C，内存 >= 2G (此处的硬件需求是最低标准，一定要保证至少2C2G)

关闭swap

1
2
$ swapoff -a  # 临时关闭，特别注意如果你是用的虚拟机，并且使用临时关闭这种方法，那么下次开机时也需要再设置一次
$ sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab   # 永久关闭

关闭防火墙和SELinux

1
2
$ systemctl stop firewalld && systemctl disable firewalld
$ sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config  && setenforce 0

内核调整,将桥接的IPv4流量传递到iptables的链

1
2
3
4
5
$ cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sysctl --system

设置系统时区并同步服务器

1
2
$ yum install -y ntpdate
$ ntpdate time.windows.com

docker安装

1
2
3
4
5
6
7
8
9
10
# 添加docker-ce YUM软件源
$ wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
# 先使用命令查看系统支持的docker版本，同时要兼顾kubernetes版本需要的docker版本
$ yum list docker-ce --showduplicates|sort -r 
# 然后安装对应版本的docker，本文选择17.09.0版本
$ yum install docker-ce-17.09.0.ce -y
# 启动docker并设置为开机自启
$ systemctl enable docker && systemctl start docker
# 查看docker版本
$ docker --version

注意：安装完毕启动docker时，可能会报如下错误：

1
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.

解决方式：

1
2
$ vim /etc/docker/daemon.json
添加内容：{"storage-driver":"devicemapper"}，保存，重新启动docker即可。

添加kubernetes YUM软件源

1
2
3
4
5
6
7
8
9
$ cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装kubeadm、kubelet和kubectl

1
2
3
$ yum install -y kubelet-1.16.3 kubeadm-1.16.3 kubectl-1.16.3
# 配置kubelet为开机自启
$ systemctl enable kubelet

2. 部署Kubernetes Master

只需要在Master 节点执行，这里的apiserver地址需要修改成自己的master地址：

1
2
3
4
5
6
$ kubeadm init \
--apiserver-advertise-address=192.168.4.116 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.16.3 \
--service-cidr=10.1.0.0/16 \
--pod-network-cidr=10.244.0.0/16

注意：由于默认拉取的镜像地址k8s.gcr.io国内无法访问，这里指定阿里云镜像仓库地址。

输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
[init] Using Kubernetes version: v1.16.3
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [bonree-zq-3 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.1.0.1 192.168.4.116]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [bonree-zq-3 localhost] and IPs [192.168.4.116 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [bonree-zq-3 localhost] and IPs [192.168.4.116 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 43.003446 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node bonree-zq-3 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node bonree-zq-3 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: c3f704.tu2gpew6v7lkg3uc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.4.116:6443 --token c3f704.tu2gpew6v7lkg3uc \
    --discovery-token-ca-cert-hash sha256:2a01ae669b7539d74f0dd7e1a29d2facc813f4b917aa16efb30dd1ac1ec27333

当看到 Your Kubernetes control-plane has initialized successfully! 这一句时表明Kubernetes Master已经初始化成功，接下来根据输出提示进行如下操作 （注意如果执行了kubeadm reset进行过kubeadm的重置 (重置时Node节点上也要执行kubeadm reset)并重新初始化master，则每次初始化完master之后都要根据提示执行如下操作）：

1
2
3
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

3. 加入Kubernetes Node

在所有Node节点上执行：

1
2
$ kubeadm join 192.168.4.116:6443 --token c3f704.tu2gpew6v7lkg3uc \
    --discovery-token-ca-cert-hash sha256:2a01ae669b7539d74f0dd7e1a29d2facc813f4b917aa16efb30dd1ac1ec27333

输出如下内容时表示join成功：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[preflight] Running pre-flight checks
	[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

4. 安装网络插件

未安装网络插件之前，如果执行 kubectl get nodes 查看所有节点的信息，则会显示NotReady，因此安装网络插件是必须的。

注意：本人在初次执行 kubectl get nodes 时报错如下：

1
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")

原因是因为我执行过kubeadm reset，然后再 kubeadm init，即重新初始化过 kubeadm，但是没有根据提示重新执行如下的三条命令，所以会出现这个问题。重新执行一下即可。

1
2
3
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

本文选择weave作为CNI网络插件：

1
2
# 如下命令适用于kubernetes1.4以后的版本
$ kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

执行之后可以验证Kubernetes集群相关的Pod是否都正常创建并运行：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   coredns-58cc8c89f4-qs8gx              1/1     Running   0          29m
kube-system   coredns-58cc8c89f4-w46vr              1/1     Running   0          29m
kube-system   etcd-bonree-zq-3                      1/1     Running   0          28m
kube-system   kube-apiserver-bonree-zq-3            1/1     Running   0          28m
kube-system   kube-controller-manager-bonree-zq-3   1/1     Running   0          28m
kube-system   kube-proxy-fmtvc                      1/1     Running   0          23m
kube-system   kube-proxy-jzbnp                      1/1     Running   0          25m
kube-system   kube-proxy-trf9v                      1/1     Running   0          29m
kube-system   kube-scheduler-bonree-zq-3            1/1     Running   0          28m
kube-system   weave-net-bnz9m                       2/2     Running   0          2m3s
kube-system   weave-net-p7bwq                       2/2     Running   0          2m3s
kube-system   weave-net-rhdfq                       2/2     Running   0          2m3s

注意：Pod创建中的STATUS为ContainerCreating，等容器创建完毕之后变为Running。

然后查看集群节点状态：

1
2
3
4
5
$ kubectl get nodes 
NAME          STATUS   ROLES    AGE   VERSION
bonree-zq-1   Ready    <none>   25m   v1.16.3
bonree-zq-2   Ready    <none>   23m   v1.16.3
bonree-zq-3   Ready    master   29m   v1.16.3

均为Ready则正常。

5. 部署 Dashboard

注意：dashboard暂时不支持v16版本，本文使用dashboardv2.0。部署Dashboard需要两个镜像，如下：

1
2
kubernetesui/dashboard   v2.0.0-beta5    b46f92585200   8 weeks ago     91.5MB
kubernetesui/metrics-scraper     v1.0.1     709901356c11     4 months ago    40.1MB

可以在执行 kubectl create -f recommended.yaml 之前提前pull下来，注意点就是可能第一个镜像会pull不下来，需要简单的配置一下，将 /etc/docker/daemon.json 文件内容补充如下，主要是添加了registry-mirrors。

1
{"storage-driver":"devicemapper","registry-mirrors": ["https://registry.docker-cn.com","http://hub-mirror.c.163.com"]}

然后重启docker：systemctl restart docker。

1.安装前先查看所有的pod详细信息：

1
$ kubectl get pods -A -o wide

2.下载并修改Dashboard脚本

1
$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta5/aio/deploy/recommended.yaml

修改recommended.yaml文件内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
---
# 增加直接访问端口
kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort #增加
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30009 #增加
  selector:
    k8s-app: kubernetes-dashboard

3.安装Dashboard

1
2
3
4
5
6
7
8
9
10
#安装
$ kubectl create -f recommended.yaml

#检查结果，当看到有以下两个pod时表示创建成功
$ kubectl get pods -A -o wide
kubernetes-dashboard   dashboard-metrics-scraper-76585494d8-vqxhj   1/1     Running   0          4m55s   10.40.0.2       bonree-zq-1   <none>           <none>
kubernetes-dashboard   kubernetes-dashboard-6b86b44f87-qxhwh        1/1     Running   0          4m55s   10.32.0.3       bonree-zq-2   <none>           <none>


$ kubectl get service -n kubernetes-dashboard  -o wide

4.创建Dashboard管理员

1
2
3
4
5
6
7
8
9
10
11
12
$ vim dashboard-admin.yaml
# 内容如下
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: dashboard-admin
  namespace: kubernetes-dashboard
  
# 保存退出后执行
$ kubectl create -f dashboard-admin.yaml

为用户分配权限：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ vim dashboard-admin-bind-cluster-role.yaml
# 内容如下
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dashboard-admin-bind-cluster-role
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: dashboard-admin
  namespace: kubernetes-dashboard
# 保存退出后执行
$ kubectl create -f dashboard-admin-bind-cluster-role.yaml

5.查看并复制用户Token

1
$ kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep dashboard-admin | awk '{print $1}')

访问 https://masterIP:30009 , 谷歌浏览器不行，推荐使用火狐，选择Token登录，输入刚才复制的密钥即可。

以上就是kubernetes的kubeadm方式部署以及可视化工具dashboard部署的全部过程及其注意事项，但是还有一个小小的瑕疵，就是在页面上没有安装metrics-server所以Pods的CPU、内存情况是看不到的。

6.安装metrics-server

heapster已经被metrics-server取代。

在Master上执行安装：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ cd /usr/local/env   # 目录自己选择

$ mkdir metrics-server

$ cd metrics-server

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/aggregated-metrics-reader.yaml

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/auth-delegator.yaml

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/auth-reader.yaml

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/metrics-apiservice.yaml

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/metrics-server-deployment.yaml

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/metrics-server-service.yaml

$ wget https://raw.githubusercontent.com/kubernetes-incubator/metrics-server/master/deploy/1.8%2B/resource-reader.yaml

修改metrics-server-deployment.yaml文件：

1
2
其中的 image: k8s.gcr.io/metrics-server-amd64:v0.3.6
修改为 registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6

因为默认的镜像是国外的，下不动。

然后执行如下步骤安装：

1
2
3
4
5
6
7
8
9
# 安装
$ kubectl create -f ../metrics-server

# 1-2分钟后查看结果，等显示了如下信息，页面上也就看到了CPU和内存的信息
$ kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
bonree-zq-1   51m          1%     1061Mi          61%       
bonree-zq-2   84m          2%     1057Mi          61%       
bonree-zq-3   223m         5%     1380Mi          80%

大功告成！

坚持原创技术分享，您的支持将鼓励我继续创作！

------本文结束 感谢您的阅读------