Install Multi-Master Kubernetes Cluster

System Setup

Setup Pre requsite

Root user access on all nodes
Stop & Disable Firewall and Selinux on all nodes
Add hostnames and IPs in /etc/hosts file on all nodes
cat >> /etc/hosts <<EOF 192.168.50.180 kube-ha.linuxtechspace.com kube-ha 192.168.50.181 kube-ha1.linuxtechspace.com kube-ha1 192.168.50.182 kube-ha2.linuxtechspace.com kube-ha2 192.168.50.183 kube-master1.linuxtechspace.com kube-master1 192.168.50.184 kube-master2.linuxtechspace.com kube-master2 192.168.50.185 kube-master3.linuxtechspace.com kube-master3 192.168.50.186 kube-worker1.linuxtechspace.com kube-worker1 192.168.50.187 kube-worker2.linuxtechspace.com kube-worker2 192.168.50.188 kube-worker3.linuxtechspace.com kube-worker3 EOF

Setup Load Balancer Nodes

Install HAProxy & Keepalive services
# yum install -y haproxy keepalived psmisc
Configure HA Proxy service

Add following entries in the configuration file.

cat >> /etc/haproxy/haproxy.cfg <<EOF frontend kubernetes bind 192.168.50.180:6443 option tcplog mode tcp default_backend kubernetes-master-nodes backend kubernetes-master-nodes mode tcp balance roundrobin option tcp-check server kube-master1.linuxtechspace.com 192.168.50.183:6443 check fall 3 rise 2 server kube-master2.linuxtechspace.com 192.168.50.184:6443 check fall 3 rise 2 server kube-master3.linuxtechspace.com 192.168.50.185:6443 check fall 3 rise 2 listen stats 192.168.50.180:8080 mode http stats enable stats uri / stats realm HAProxy\ Statistics stats auth admin:haproxy EOF systemctl enable --now haproxy
Configure keepalive service

Note: Set the following two options as per the given values:

Load Balancer 1 (kube-ha1):
state MASTER
priority 101

Load Balancer 2 (kube-ha2):
state BACKUP
priority 100

cat > /etc/keepalived/keepalived.conf <<EOF ! Configuration File for keepalived global_defs { notification_email { root@localhost } notification_email_from root@localhost smtp_server localhost smtp_connect_timeout 30 router_id LVS_DEVEL } # Script used to check if HAProxy is running vrrp_script check_haproxy { script "killall -0 haproxy" # check haproxy process interval 2 # check every 2 seconds weight 2 # add 2 points if health is OK } vrrp_instance VI_1 { state MASTER interface enp0s3 virtual_router_id 51 priority 101 advert_int 1 virtual_ipaddress { 192.168.50.180/24 } track_scripts { check_haproxy } } EOF systemctl enable --now keepalived
Start & Enable keepalive service
# systemctl enable --now keepalived
Check that VIP (192.168.50.180) is now successfully setup on master load balancer node kube-lb1
# ip addr show dev enp0s3 2: enp0s3: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 08:00:27:22:26:a6 brd ff:ff:ff:ff:ff:ff inet 192.168.50.181/24 brd 192.168.50.255 scope global noprefixroute enp0s3 valid_lft forever preferred_lft forever inet 192.168.50.180/24 scope global secondary enp0s3 valid_lft forever preferred_lft forever

Setup Master and Worker Nodes

Create non-privileged user kadmin and grant SUDO access to it
useradd kadmin echo "kadmin ALL=(ALL) NOPASSWD:ALL" | tee -a /etc/sudoers
Disable SWAP
swapoff -a sed -i '/swap/s/^/#/' /etc/fstab
Add Netfilter module
modprobe br_netfilter echo "br_netfilter" | tee /etc/modules-load.d/k8s.conf
Setup Kernel Parameters
cat >> /etc/sysctl.conf <<EOF net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl --system
Install Container Runtime containerd
cat > /etc/yum.repos.d/docker-ce.repo <<EOF [docker-ce-stable] name=Docker CE Stable - \$basearch baseurl=https://download.docker.com/linux/centos/\$releasever/\$basearch/stable enabled=1 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [centos-extra] name=CentOS extra baseurl=http://mirror.centos.org/centos/7/extras/x86_64/ enabled=1 gpgcheck=0 EOF yum -y install containerd.io sed -i '/disabled_plugins/s/^/#/' /etc/containerd/config.toml systemctl restart containerd systemctl enable containerd
Install kubernetes
cat > /etc/yum.repos.d/kubernetes.repo <<EOF [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch enabled=1 gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude=kubelet kubeadm kubectl EOF yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes systemctl enable --now kubelet

Create Kubernetes Cluster

Note: Perform this step only on first master node kube-master1

Login as kadmin user
su - kadmin
Create kubernetes cluster
sudo kubeadm init --control-plane-endpoint=192.168.50.180 --upload-certs --pod-network-cidr=192.168.0.0/16 --apiserver-advertise-address=192.168.50.183
Setup cluster configuration for kadmin user
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
Install POD Network Calito
kubectl create -f https://docs.projectcalico.org/manifests/calico.yaml
Check status of Nodes and Pods
kubectl get nodes kubectl get pods -A

Join Additional Master Nodes to Cluster

Note: Perform these steps on remaining two Master nodes.

Login as kadmin user
su - kadmin
Join the Cluster as Master
sudo kubeadm join 192.168.50.180:6443 --token jxn1vv.psew0gy9ichtyuhg \ --discovery-token-ca-cert-hash sha256:118f38284018461121767962629f2aed68aa4ac4bc3bbece9183a5aa926db404 \ --control-plane --certificate-key f9f1e506542a8697b5a6aedb04c83b08ebd09da289ac5503a81477a94bb2b795 \ --apiserver-advertise-address=192.168.50.184

Join Worker Nodes to Cluster

Note: Perform these steps on all three worker nodes

Login as kadmin user
su - kadmin
Login as kadmin user
sudo kubeadm join 192.168.50.180:6443 --token jxn1vv.psew0gy9ichtyuhg \ --discovery-token-ca-cert-hash sha256:118f38284018461121767962629f2aed68aa4ac4bc3bbece9183a5aa926db404

Check Cluster Status

Note: Perform this step on any of the master node

Check cluster nodes status
kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME kube-master1.linuxtechspace.com Ready control-plane 46m v1.24.0 192.168.50.183 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.el7.x86_64 containerd://1.6.4 kube-master2.linuxtechspace.com Ready control-plane 40m v1.24.0 192.168.50.184 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.el7.x86_64 containerd://1.6.4 kube-master3.linuxtechspace.com Ready control-plane 26m v1.24.0 192.168.50.185 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.el7.x86_64 containerd://1.6.4 kube-worker1.linuxtechspace.com Ready 10m v1.24.0 192.168.50.186 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.el7.x86_64 containerd://1.6.4 kube-worker2.linuxtechspace.com Ready 118s v1.24.0 192.168.50.187 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.el7.x86_64 containerd://1.6.4 kube-worker3.linuxtechspace.com Ready 113s v1.24.0 192.168.50.188 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.el7.x86_64 containerd://1.6.4
Check cluster pods status
kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system calico-kube-controllers-56cdb7c587-ngfr2 0/1 ContainerCreating 0 45m kube-master3.linuxtechspace.com kube-system calico-node-2r5wd 1/1 Running 1 (24m ago) 26m 192.168.50.185 kube-master3.linuxtechspace.com kube-system calico-node-4ghdg 0/1 Running 6 (2m2s ago) 10m 192.168.50.186 kube-worker1.linuxtechspace.com kube-system calico-node-dqqvt 1/1 Running 4 (35m ago) 40m 192.168.50.184 kube-master2.linuxtechspace.com kube-system calico-node-r5dxm 0/1 Running 0 119s 192.168.50.188 kube-worker3.linuxtechspace.com kube-system calico-node-sltvr 0/1 Running 0 2m5s 192.168.50.187 kube-worker2.linuxtechspace.com kube-system calico-node-tcgdz 1/1 Running 0 45m 192.168.50.183 kube-master1.linuxtechspace.com kube-system coredns-6d4b75cb6d-9cw9s 1/1 Running 0 46m 192.168.37.130 kube-master1.linuxtechspace.com kube-system coredns-6d4b75cb6d-qrvg2 1/1 Running 0 46m 192.168.37.129 kube-master1.linuxtechspace.com kube-system etcd-kube-master1.linuxtechspace.com 1/1 Running 0 46m 192.168.50.183 kube-master1.linuxtechspace.com kube-system etcd-kube-master2.linuxtechspace.com 1/1 Running 0 39m 192.168.50.184 kube-master2.linuxtechspace.com kube-system etcd-kube-master3.linuxtechspace.com 1/1 Running 0 26m 192.168.50.185 kube-master3.linuxtechspace.com kube-system kube-apiserver-kube-master1.linuxtechspace.com 1/1 Running 0 46m 192.168.50.183 kube-master1.linuxtechspace.com kube-system kube-apiserver-kube-master2.linuxtechspace.com 1/1 Running 0 39m 192.168.50.184 kube-master2.linuxtechspace.com kube-system kube-apiserver-kube-master3.linuxtechspace.com 1/1 Running 0 26m 192.168.50.185 kube-master3.linuxtechspace.com kube-system kube-controller-manager-kube-master1.linuxtechspace.com 1/1 Running 1 (40m ago) 46m 192.168.50.183 kube-master1.linuxtechspace.com kube-system kube-controller-manager-kube-master2.linuxtechspace.com 1/1 Running 0 39m 192.168.50.184 kube-master2.linuxtechspace.com kube-system kube-controller-manager-kube-master3.linuxtechspace.com 1/1 Running 0 25m 192.168.50.185 kube-master3.linuxtechspace.com kube-system kube-proxy-9w4wv 1/1 Running 0 10m 192.168.50.186 kube-worker1.linuxtechspace.com kube-system kube-proxy-bz4pl 1/1 Running 0 26m 192.168.50.185 kube-master3.linuxtechspace.com kube-system kube-proxy-llg2b 1/1 Running 0 46m 192.168.50.183 kube-master1.linuxtechspace.com kube-system kube-proxy-mz7fv 1/1 Running 0 2m5s 192.168.50.187 kube-worker2.linuxtechspace.com kube-system kube-proxy-tzlj7 1/1 Running 0 119s 192.168.50.188 kube-worker3.linuxtechspace.com kube-system kube-proxy-vh2kw 1/1 Running 0 40m 192.168.50.184 kube-master2.linuxtechspace.com kube-system kube-scheduler-kube-master1.linuxtechspace.com 1/1 Running 1 (40m ago) 46m 192.168.50.183 kube-master1.linuxtechspace.com kube-system kube-scheduler-kube-master2.linuxtechspace.com 1/1 Running 0 38m 192.168.50.184 kube-master2.linuxtechspace.com kube-system kube-scheduler-kube-master3.linuxtechspace.com 1/1 Running 0 25m 192.168.50.185 kube-master3.linuxtechspace.com