raynix 筆記

How I Upgraded Sail Operator And How I Would Do It Next Time

It’s been a while since I installed Istio with Sail Operator in my home lab Kubernetes cluster. The version of Sail Operator was 0.2 and now 1.26 is available. There are so many releases between 0.2 and 1.26! Can I upgrade directly to the latest and skip all the middle ones? It’s my home lab so no one else is gonna stop me.

Since everything is super automated, I only need to change a version to get the upgrade done and so I did. After I pushed the changes, I saw a few CRDs(Custom Resource Definition) got stuck in ArgoCD. I didn’t think too much because it’s just my home lab and forced the deletion of old CRDs by deleting the finalizers. And from a hind sight, this was the big mistake.

After all CRDs are refreshed with the latest version, ArgoCD went dark and errors are everywhere:

Error from server: request to convert CR from an invalid group/version: sailoperator.io/v1alpha1

I took a closer look. The new version has an API version of sailoperator.io/v1 and the old one has sailoperator.io/v1alpha1. Now the Kubernetes API server refused to do anything related to Sail Operator resources since the old API group v1alpha1 doesn’t exist anymore. The issue happens because:

  1. I still have the resources in the cluster with the old API version. I though ArgoCD will delete the old one cleverly but it doesn’t 🙂
  2. This was why the old CRDs were stuck – the finalizers were protecting existing resources in use but I forced the deletion!

In order to delete the old custom resources I have to re-instate the old CRDs:

  1. I set ArgoCD to manual mode, in case the auto sync messes up the order
  2. I wiped the sail-operator namespace along with the new CRDs
  3. Then I reverted my git commit so ArgoCD will bring back the 0.2 version of Sail Operator
  4. It’s time to delete all the old custom resources!
  5. Revert the revert and install Sail Operator 1.26
  6. Install Istio 1.26!

Finally I got everything green again. Next time, I’ll think twice when deleting stuff blocked by finalizers 🙂

BTW, I didn’t break the gateway so web access to my home lab including this blog and ArgoCD, etc. was going uninterrupted.

admin

,