Pre-Requisite
- Three Linux Machines/VMs (Minimum Specs:
2 cores, 4GB RAM, 16GB Hard Disk) - Can communicate with each other (Preferred to be on Same LAN)
- Open ports: 22 (SSH), 6443 (Kubernetes API), 2379-2380 (etcd), 10250 (kubelet)
2 cores, 4GB RAM, 16GB Hard Disk)Linux Server Preparation for Kubernetest Cluster
Update system and install prerequisites
apt update && apt upgrade -y
apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates chronyDisable swap (Kubernetes requirement)
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstabEnable kernel modules and sysctl params
cat <<EOF | tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOFLoad kernel Modules
modprobe overlay
modprobe br_netfilterset system variables
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOFLoad system variables
sysctl --systemInstall Docker and Kubernetes
Note
Execute the following steps on all Three Nodes.
Note
Execute the following steps on all Three Nodes.
Add Docker Repo
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian bookworm stable" | tee /etc/apt/sources.list.d/docker.listUpdate and install Docker
apt update
apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-pluginConfigure containerd
mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.tomlSet cgroup driver to systemd
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.tomlRestart containerd
systemctl restart containerd
systemctl enable containerdAdd Kubernetes repo
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.listInstall kubelet, kubeadm, kubectl
apt update
apt install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectlInitiate Kubernetes Cluster
Note
Execute the following commands only on Master Nodes.
Note
Execute the following commands only on Master Nodes.
Initialize cluster with Calico network (recommended for Debian)
kubeadm initOR if want to define CIDR do it as below
kubeadm init \
--pod-network-cidr=192.168.100.0/24 \
--service-cidr=192.168.101.0/24 # Example: Different CIDR for servicesOUTPUT of the above command will guide to executethe further commands
I0522 10:17:11.162566 18880 version.go:256] remote version is much newer: v1.33.1; falling back to: stable-1.28
[init] Using Kubernetes version: v1.28.15
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0522 10:17:19.988252 18880 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.8" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local kubmaster] and IPs [10.96.0.1 192.168.10.6]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubmaster localhost] and IPs [192.168.10.6 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubmaster localhost] and IPs [192.168.10.6 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 6.002958 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kubmaster as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node kubmaster as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: oi42uq.qkhghzhbgbekm81a
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.10.6:6443 --token oi42uq.qkhghzhbgbekm81a --discovery-token-ca-cert-hash sha256:353a1ad9271fdfc883cf2a7fa47b232580dd10a5fc3dcce11377d79daeed7518
Copy commands from the OUTPUT
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown (id -g) $HOME/.kube/config
kubeadm join 192.168.10.6:6443 --token oi42uq.qkhghzhbgbekm81a --discovery-token-ca-cert-hash sha256:353a1ad9271fdfc883cf2a7fa47b232580dd10a5fc3dcce11377d79daeed7518
Now Execute it in the following order
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configAfter that Install Calico network plugin
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/tigera-operator.yaml
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/custom-resources.yaml
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yamlNow Join Master node to cluster
kubeadm join 192.168.10.6:6443 --token oi42uq.qkhghzhbgbekm81a --discovery-token-ca-cert-hash sha256:353a1ad9271fdfc883cf2a7fa47b232580dd10a5fc3dcce11377d79daeed7518Now Execute the following commands
kubectl get nodes -o wideOUTPUT
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubmaster Ready control-plane 5h26m v1.28.15 192.168.10.6 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-32-cloud-amd64 containerd://1.7.27
confirm coreDNS IP as well.
kubectl get svc -n kube-system | grep -E 'kube-dns|coredns'OUTPUT
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3m2s
Add node to Kubernetes Cluster
Important
Please make sure that follwoing ports are reachable TCP/6443 Try telnel port 6443
apt install telnet
telnet 192.168.10.8 6443
Trying 192.168.10.8...
Connected to 192.168.10.8.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
Note
Execute the following commands only on Worker Nodes.
Important
Please make sure that follwoing ports are reachable TCP/6443 Try telnel port 6443
apt install telnet
telnet 192.168.10.8 6443
Trying 192.168.10.8...
Connected to 192.168.10.8.
Escape character is '^]'.
^]
telnet> quit
Connection closed.Note
Execute the following commands only on Worker Nodes.
Go to each Node and execute the following command
kubeadm join 192.168.10.6:6443 --token oi42uq.qkhghzhbgbekm81a --discovery-token-ca-cert-hash sha256:353a1ad9271fdfc883cf2a7fa47b232580dd10a5fc3dcce11377d79daeed7518Execute the follwoing command again on Master Node
kubectl get nodes -o wideOUTPUT
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubmaster Ready control-plane 5h26m v1.28.15 192.168.10.6 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-32-cloud-amd64 containerd://1.7.27
kubnode1 Ready <none> 5h3m v1.28.15 192.168.10.5 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-35-cloud-amd64 containerd://1.7.27
kubnode2 Ready <none> 5h2m v1.28.15 192.168.10.7 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-35-cloud-amd64 containerd://1.7.27
Troubleshooting
Commands which can help in troubleshooting Check Logs
journalctl -u kubelet -fCheck if system pods are working fine
kubectl get pods -n kube-system OUTPUT
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-658d97c59c-gvn5s 1/1 Running 0 5h21m
calico-node-hw42j 1/1 Running 0 5h21m
calico-node-p55zs 1/1 Running 0 5h11m
calico-node-qdwlh 1/1 Running 0 5h12m
coredns-5dd5756b68-r85lr 1/1 Running 0 5h35m
coredns-5dd5756b68-zxsss 1/1 Running 0 5h35m
etcd-kubmaster 1/1 Running 0 5h35m
kube-apiserver-kubmaster 1/1 Running 0 5h35m
kube-controller-manager-kubmaster 1/1 Running 0 5h35m
kube-proxy-cllll 1/1 Running 0 5h12m
kube-proxy-pkxcp 1/1 Running 0 5h11m
kube-proxy-s5c2c 1/1 Running 0 5h35m
kube-scheduler-kubmaster 1/1 Running 0 5h35m
Check if thee are any errors in containerd
systemctl status containerd
crictl ps # Verify container runtimeHow to Reset Master Node
Drain all Worker Nodes
for node in $(kubectl get nodes -o name | grep -v master); do
kubectl drain $node --ignore-daemonsets --delete-emptydir-data
doneStop Control Plane Services
sudo kubeadm reset -f
sudo systemctl stop kubelet
sudo systemctl stop dockerHow to Reset a Worker Node
sudo kubeadm reset
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/
sudo rm -rf /var/lib/etcd/
sudo rm -rf $HOME/.kube
sudo iptables -F
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -X
sudo systemctl stop kubelet
sudo systemctl disable kubelet
sudo systemctl stop docker || sudo systemctl stop containerd
sudo systemctl daemon-reload
sudo kubeadm reset
sudo rm -rf /etc/kubernetes/
sudo rm -rf /var/lib/kubelet/
sudo rm -rf /var/lib/etcd/
sudo rm -rf $HOME/.kube
sudo iptables -F
sudo iptables -t nat -F
sudo iptables -t mangle -F
sudo iptables -X
sudo systemctl stop kubelet
sudo systemctl disable kubelet
sudo systemctl stop docker || sudo systemctl stop containerd
sudo systemctl daemon-reloadconfirm that there is no servce listening on port 10250
sudo netstat -tulnp | grep 10250For Next Steps use this link
Enjoy 😉
No comments:
Post a Comment