HTTP 525/526 errors are unofficial HTTP errors specific to CloudFlare. In a nut shell they mean HTTP TLS cert related errors when communicating with origin web servers. I got one of these from my sites so I took a look into this.

I use cert-manager’s integration with Let’s Encrypt to automatically renew my TLS certs. So cert-manager was my first destination. I saw strange errors from the cert-manager server saying something wrong with CloudFlare API, more importantly my cert-manager hasn’t been upgraded for 2 years. So I decided to upgrade cert-manager first.

The upgrade went well except that I didn’t have access to my ArgoCD server, so I used a port forwarding:

k port-forward argocd-server-xxx 8080:8080
# open http://localhost:8080 in a browser

The existing certs seemed to be stuck though, but that’s an easy fix – I deleted them and let ArgoCD re-create. New certs reached READY state shortly. But strangely the 525 error still persisted, which left me to suspect that the new certs were not picked up by the ingress gateway. As soon as I restarted the gateway using

k rollout restart deploy kubernetes-gateway

the error went away. 🙂