Solved: Uninstallation of config-connector Got Stuck in ArgoCD

The Kubernetes Config Connector is another level of IaC(Infrastructure as Code): It wraps Google Cloud resources like a Cloud Load Balancer with Kubernetes CRDs(Custom Resource Definition) so instead of writing Terraform HCL I can write YAML to manage GCP infrastructure.

However when there’s a need to uninstall a config-connector, it got stuck in ArgoCD

As always, kubectl describe is my friend in this kind of situations.

$ k describe ns config-connector-ops
Name:         config-connector-ops
Annotations:  <none>
Status:       Terminating
  Type                                         Status  LastTransitionTime               Reason                Message
  ----                                         ------  ------------------               ------                -------
  NamespaceDeletionDiscoveryFailure            False   Tue, 11 Apr 2023 00:30:43 +1000  ResourcesDiscovered   All resources successfully discovered
  NamespaceDeletionGroupVersionParsingFailure  False   Thu, 06 Apr 2023 11:03:48 +1000  ParsedGroupVersions   All legacy kube types successfully parsed
  NamespaceDeletionContentFailure              False   Thu, 06 Apr 2023 11:03:48 +1000  ContentDeleted        All content successfully deleted, may be waiting on finalization
  NamespaceContentRemaining                    True    Thu, 06 Apr 2023 11:03:48 +1000  SomeResourcesRemain   Some resources are remaining: has 1 resource instances, has 2 resource instances
  NamespaceFinalizersRemaining                 True    Thu, 06 Apr 2023 11:03:48 +1000  SomeFinalizersRemain  Some content in the namespace has finalizers remaining: in 3 resource instances

It’s a bit obvious already: a few resources in this namespace has finalizer and that’s why they are still pending termination. The first one to take a closer look at is the configconnectorcontext resource as shown in the screen shot.

$ k describe
    error during reconciliation: error building deployment objects: error transforming namespaced components: error getting namespace id for namespace config-connector-ops: error creating configmap 'configconnector-operator-system/namespace-id': configmaps "namespace-id" is forbidden: unable to create new content in namespace configconnector-operator-system because it is being terminated

I guess this is caused by ArgoCD trying to delete all stuff all at once, so the context object couldn’t change a config-map because the namespace was also marked for deletion. I had to manually remove finalizers in these stuck resources by manually editing them:

k edit
k edit configconnector
k edit cnrm-manager-ns-binding-config-connector-ops
k edit cnrm-admin-binding-config-connector-ops

Then ArgoCD was glad to tell me that the config-connector app was successfully delete. I also did the following clean-up as ArgoCD missed these:

k delete
k delete
k delete
k delete cnrm-manager-cluster-binding-config-connector-ops