Recently I had to renew expired kubernetes certificates on my home lab cluster after getting locked out from managing it. I wasn’t tracking their age and all of a sudden I found them expired. Kubeadm has a feature to auto-renew certificates during control plane upgrades. Unfortunately I had not done an upgrade on my home cluster in the last year. I realised when issuing a kubectl command to the cluster and receiving an error along the lines of
x509: certificate has expired or is not yet valid.
To regain access I needed to SSH onto a master node in the cluster and do the following:
Move / backup old certificate and kubeadm config files:
sudo mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old sudo mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old sudo mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old sudo mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old sudo mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old sudo mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old sudo mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old sudo mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old sudo mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old sudo mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old
Renew Expired Kubernetes Certificates
kubeadm to renew all certificates.
sudo kubeadm alpha certs renew all
Inspect the generated certificates under /etc/kubernetes/pki to make sure they are generated correctly. E.g. expiry dates a year in the future, etc… For example, checking the new kube api server certificate details:
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text -certopt no_subject,no_version,no_serial,no_signame,no_issuer,no_pubkey,no_sigdump,no_header,no_aux
Regenerate Kubeconfig Files
Now you can regenerate the kubeconfig files for the kubelet, controller-manager, scheduler, etc…
sudo kubeadm alpha phase kubeconfig all --apiserver-advertise-address 10.0.0.50
Note: you can leave the
--apiserver-advertise-address option off, but the value used will default to your system’s default network interface.
Check that all the .conf files under /etc/kubernetes are re-created. i.e.
You can also update your kubectl config with the admin.conf file that was generated if that is the config/context you were using from the master node.
mv .kube/config .kube/config.old sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config sudo chmod 777 $HOME/.kube/config export KUBECONFIG=.kube/config
If your kube services have stopped on the master (kubelet etc) as a result of expired certificates and having tried to restart services recently (as was my case), reboot the master node at this point. All services that were failing to start because of expired certificates should now start correctly.
Reconnect and check everything is working as expected:
kubectl get nodes kubeadm token list