When upgrading my home-lab Kubernetes cluster from 1.32 to 1.35, I encountered some error that the first node upgraded became not ready, eg.
k get nodes NAME STATUS ROLES AGE VERSION kmaster2 Ready control-plane 2y283d v1.35.4 knode3 NotReady <none> 2y282d v1.32.4
It should show 1.35 but it didn’t. I guess there should be some error in the logs of the kubelet service.
# ssh to knode3 and become root journalctl -f -u kubelet ... May 12 11:58:55 knode3 kubelet[2784421]: E0512 11:58:55.346466 2784421 run.go:72] "command failed" err="failed to parse kubelet flag: unknown flag: --pod-infra-container-image" May 12 11:58:55 knode3 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE May 12 11:58:55 knode3 systemd[1]: kubelet.service: Failed with result 'exit-code'.
So this --pod-infra-container-image flag needs to be removed. Easy.
vim /var/lib/kubelet/kubeadm-flags.env # then remove the flag found in the file then restart it systemctl restart kubelet # from my laptop k get nodes NAME STATUS ROLES AGE VERSION kmaster2 Ready control-plane 2y283d v1.35.4 knode3 Ready <none> 2y282d v1.35.4 ...
I probably need to do this for every node but I guess this is just a one-off 🙂