setenforce 0 sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux sed -i "s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config
#若问题 执行sysctl -p 时出现: sysctl -p sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory 解决方法: modprobe br_netfilter ls /proc/sys/net/bridge
[root@etcd-host1 k8s]# systemctl status keepalived ● keepalived.service - LVS and VRRP High Availabilitymonitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2018-01-19 10:27:58 CST; 8h ago Main PID: 1158 (keepalived) CGroup: /system.slice/keepalived.service ├─1158 /usr/sbin/keepalived -D ├─1159 /usr/sbin/keepalived -D └─1161 /usr/sbin/keepalived -D
Jan 19 10:28:00 etcd-host1 Keepalived_vrrp[1161]: Sending gratuitous ARP on ens32 for 10.129.6.220 Jan 19 10:28:05 etcd-host1 Keepalived_vrrp[1161]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens32 for 10.129.6.220
[root@node02 ~]# etcdctl --endpoints=https://${NODE_IP}:2379 --ca-file=/etc/etcd/ssl/ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem cluster-health member 18699a64c36a7e7b is healthy: got healthy result from https://10.129.6.213:2379 member 5dbd6a0b2678c36d is healthy: got healthy result from https://10.129.6.211:2379 member 6b1bf02f85a9e68f is healthy: got healthy result from https://10.129.6.212:2379 cluster is healthy
[root@etcd-host1 k8s]# kubectl get node NAME STATUS ROLES AGE VERSION etcd-host1 noReady master 5h v1.9.0 [root@etcd-host1 k8s]# kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"}
问题记录
1 2 3 4 5 6 7 8 9 10 11
如果使用kubeadm初始化集群,启动过程可能会卡在以下位置,那么可能是因为cgroup-driver参数与docker的不一致引起 [apiclient] Created API client, waiting for the control plane to become ready journalctl -t kubelet -S '2017-06-08'查看日志,发现如下错误 error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "systemd" 需要修改KUBELET_CGROUP_ARGS=--cgroup-driver=systemd为KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs
vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf #Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
#观察当Master01主节点关闭后,被节点VIP状态 BACKUP 切换到 MASTER [root@etcd-host2 net.d]# systemctl status keepalived ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2018-01-22 13:54:17 CST; 21s ago
Jan 22 13:54:17 etcd-host2 Keepalived_vrrp[15908]: VRRP_Instance(VI_1) Transition to MASTER STATE Jan 22 13:54:17 etcd-host2 Keepalived_vrrp[15908]: VRRP_Instance(VI_1) Received advert with higher priority 120, ours 110 Jan 22 13:54:17 etcd-host2 Keepalived_vrrp[15908]: VRRP_Instance(VI_1) Entering BACKUP STATE
#切换到 MASTER [root@etcd-host2 net.d]# systemctl status keepalived ● keepalived.service - LVS and VRRP High Availability Monitor Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2018-01-22 13:54:17 CST; 4min 6s ago Jan 22 14:03:02 etcd-host2 Keepalived_vrrp[15908]: VRRP_Instance(VI_1) Transition to MASTER STATE Jan 22 14:03:03 etcd-host2 Keepalived_vrrp[15908]: VRRP_Instance(VI_1) Entering MASTER STATE Jan 22 14:03:03 etcd-host2 Keepalived_vrrp[15908]: VRRP_Instance(VI_1) setting protocol VIPs. Jan 22 14:03:03 etcd-host2 Keepalived_vrrp[15908]: Sending gratuitous ARP on ens32 for 10.129.6.220
#观察 master01 关机后状态变成NotReady [root@etcd-host3 ~]# while true; do sleep 1; kubectl get node;date; done Tue Jan 22 14:03:16 CST 2018 NAME STATUS ROLES AGE VERSION etcd-host1 Ready master 19m v1.9.0 etcd-host2 Ready master 3d v1.9.0 etcd-host3 Ready master 3d v1.9.0 Tue Jan 22 14:03:17 CST 2018 NAME STATUS ROLES AGE VERSION etcd-host1 NotReady master 19m v1.9.0 etcd-host2 Ready master 3d v1.9.0 etcd-host3 Ready master 3d v1.9.0
#恢复Master主节点后,出现VIP偏移过来,api恢复 The connection to the server 10.129.6.220:6443 was refused - did you specify the right host or port? Tue Jan 23 14:14:05 CST 2018 The connection to the server 10.129.6.220:6443 was refused - did you specify the right host or port? Tue Jan 23 14:14:07 CST 2018 Tue Jan 23 14:14:18 CST 2018 NAME STATUS ROLES AGE VERSION etcd-host1 NotReady master 29m v1.9.0 etcd-host2 Ready master 3d v1.9.0 etcd-host3 Ready master 3d v1.9.0 Tue Jan 23 14:14:20 CST 2018 NAME STATUS ROLES AGE VERSION etcd-host1 Ready master 29m v1.9.0 etcd-host2 Ready master 3d v1.9.0 etcd-host3 Ready master 3d v1.9.0
参观文档
1 2 3 4 5 6 7 8 9
#k8s 官方文档 https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/ #kubeadm ha 项目文档 https://github.com/indiketa/kubeadm-ha https://github.com/cookeem/kubeadm-ha/blob/master/README_CN.md https://medium.com/@bambash/ha-kubernetes-cluster-via-kubeadm-b2133360b198 #kubespray 之前的kargo ansible项目 https://github.com/kubernetes-incubator/kubespray/blob/master/docs/ha-mode.md #若有问题或转载请注明出处 By Zeming