2. Setup Kubernetes#
Note
In the following guide, the Control Plane and Worker Node stand respectively for production machines iis-binderhub-1 and iis-binderhub-2~iis-binderhub-4. Please finish the tasks for both Control Plane and Worker Node.
2.1. Disable swap#
sudo sed -i '/swap/d' /etc/fstab
sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab
sudo swapoff -a
2.2. Disable firewall#
sudo systemctl disable --now ufw
sudo systemctl disable --now ufw
2.3. Setup bridge traffic#
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
2.4. Setup NFS Storage#
Note
For the NFS_IP, WORKER_IP_1, WORKER_IP_2, and WORKER_IP_3,
please refer to the Hardware and software.
In the iis-binderhub-1 machine (we reuse it to setup the NFS storage to share data between servers):
sudo apt update && sudo apt install nfs-kernel-server
sudo mkdir /var/nfs/general -p
sudo chown nobody:nogroup /var/nfs/general
Modify the /etc/exports:
/etc/exports#
# Replace the WORKER_IP_1, WORKER_IP_2, and WORKER_IP_3
/var/nfs/general WORKER_IP_1(rw,sync,no_subtree_check) WORKER_IP_2(rw,sync,no_subtree_check) WORKER_IP_3(rw,sync,no_subtree_check)
sudo systemctl restart nfs-kernel-server
In the worker nodes (the production machines from iis-binderhub-2 to iis-binderhub-4):
sudo apt update && sudo apt install nfs-common
sudo mkdir -p /nfs/general
sudo mount NFS_IP:/var/nfs/general /nfs/general
Modify the /etc/fstab:
/etc/fstab#
# Replace the NFS_IP
NFS_IP:/var/nfs/general /nfs/general nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
2.5. Install containerd#
sudo apt install apt-transport-https ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install containerd.io
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd
sudo apt install apt-transport-https ca-certificates curl gnupg lsb-release
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install containerd.io
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd
2.6. Install Kubernetes#
sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install kubeadm kubelet kubectl
sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install kubeadm kubelet kubectl
2.7. Initialize Kubernetes cluster#
In the Control Plane (the production machine iis-binderhub-1 with IP address CP_IP):
sudo kubeadm config images pull
sudo kubeadm init --apiserver-advertise-address=CP_IP --pod-network-cidr=10.244.0.0/16
Record the command provided by the initial command:
kubeadm join CP_IP:6443 --token my-secret-token \
--discovery-token-ca-cert-hash sha256:my-secret-hash
Create regular user configuration for Kubernetes:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
2.8. Install Container Network Interface (CNI)#
In the Control Plane (the production machine iis-binderhub-1):
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
Check pod status:
kubectl get pods -n kube-flannel
Console output#
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-f8cgz 1/1 Running 0 2m30s
2.9. Join the worker nodes#
Attention
Please ensure that you complete all tasks related to the worker node.
In the worker nodes (the production machines from iis-binderhub-2 to iis-binderhub-4),
run the command provided by the initial command:
sudo kubeadm join CP_IP:6443 --token my-secret-token \
--discovery-token-ca-cert-hash sha256:my-secret-hash
2.10. Install Load Balancer#
In the Control Plane (the production machine iis-binderhub-1):
MetalLB_RTAG=$(curl -s https://api.github.com/repos/metallb/metallb/releases/latest|grep tag_name|cut -d '"' -f 4|sed 's/v//')
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v$MetalLB_RTAG/config/manifests/metallb-native.yaml
Check pod status:
kubectl get pods -n metallb-system
Console output#
NAME READY STATUS RESTARTS AGE
controller-86f5578878-btx4m 1/1 Running 0 80m
speaker-87m8t 1/1 Running 0 80m
speaker-l49d7 1/1 Running 0 2m43s
Create a file called ipaddress_pools.yaml:
Note
For the CP_IP, WORKER_IP_1, WORKER_IP_2, and WORKER_IP_3,
please refer to the Hardware and software.
ipaddress_pools.yaml#
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: production
namespace: metallb-system
spec:
addresses: # at least one address for ingress-nginx
- CP_IP/32
- WORKER_IP_1/32
- WORKER_IP_2/32
- WORKER_IP_3/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: l2-advert
namespace: metallb-system
kubectl apply -f ipaddress_pools.yaml
2.11. Install Helm#
In the Control Plane (the production machine iis-binderhub-1):
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
2.12. Install NFS provisioner#
In the Control Plane (the production machine iis-binderhub-1 with IP address NFS_IP):
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
--set nfs.server=NFS_IP \
--set nfs.path=/var/nfs/general
kubectl patch storageclass nfs-client -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Check pod status:
kubectl get pods -n default
Console output#
NAME READY STATUS RESTARTS AGE
nfs-subdir-external-provisioner-7fd7884cbc-lt7v8 1/1 Running 0 57s