A Cluster IP is an IP address allocated in the cluster’s virtual LAN, usually allocated to Kubernetes Services. As a user nobody should care about which cluster IP a service would get because we will use the service-name.namespace-name DNS name which will map to that IP automatically.

In my case, some services in a GKE cluster got identical cluster IPs which caused them unreachable. With help from Google’s GCP support, the result is a bit funny:

The cluster was built with a /23 sub-net, which can only hosts about 510 IPs. At the time when the issue occurred, there were about 540 services in the cluster, so the last few services got duplicated IPs. It’s unexpected to me because I would rather get some error when Cluster IPs run out.

There’s no real fix to this situation as the sub-net is immutable after the cluster is built. So I migrated some workloads out of this cluster to get the service count back down.

Verdict: A Kubernetes cluster should be built with at least a /16(65534 IPs) sub-net to have a large enough IP pool for services in a busy cluster.