Orphaned Resources After Cluster Delete

[danehans]

After running the following commands, I see orphaned resources that were created by the gce ingress controller:

$ kubectl delete ing $ING -n $ING_NAMESPACE
$ gcloud container clusters delete $CLUSTER_NAME

List of orphaned resources:

$ gcloud compute instance-groups list
NAME                      LOCATION    SCOPE  NETWORK  MANAGED  INSTANCES
k8s-ig--6c90c323237f439e  us-west1-a  zone   default  No       0
k8s-ig--71cb3990c3cf10a6  us-west1-a  zone   default  No       0

$ gcloud compute backend-services list
NAME                            BACKENDS                                            PROTOCOL
k8s-be-30286--71cb3990c3cf10a6  us-west1-a/instanceGroups/k8s-ig--71cb3990c3cf10a6  HTTP
k8s-be-31225--6c90c323237f439e  us-west1-a/instanceGroups/k8s-ig--6c90c323237f439e  HTTP
k8s-be-31253--6c90c323237f439e  us-west1-a/instanceGroups/k8s-ig--6c90c323237f439e  HTTP
k8s-be-31345--71cb3990c3cf10a6  us-west1-a/instanceGroups/k8s-ig--71cb3990c3cf10a6  HTTP
k8s-be-31823--6c90c323237f439e  us-west1-a/instanceGroups/k8s-ig--6c90c323237f439e  HTTP
k8s-be-32421--71cb3990c3cf10a6  us-west1-a/instanceGroups/k8s-ig--71cb3990c3cf10a6  HTTP

$ gcloud compute url-maps list
NAME                                             DEFAULT_SERVICE
k8s-um-fortio-fortio-ingress--6c90c323237f439e   backendServices/k8s-be-31253--6c90c323237f439e
k8s-um-fortio-fortio-ingress--71cb3990c3cf10a6   backendServices/k8s-be-30286--71cb3990c3cf10a6
k8s-um-fortio-fortio-ingress2--6c90c323237f439e  backendServices/k8s-be-31823--6c90c323237f439e

$ gcloud compute target-http-proxies list
NAME                                             URL_MAP
k8s-tp-fortio-fortio-ingress--71cb3990c3cf10a6   k8s-um-fortio-fortio-ingress--71cb3990c3cf10a6
k8s-tp-fortio-fortio-ingress2--6c90c323237f439e  k8s-um-fortio-fortio-ingress2--6c90c323237f439e

$ gcloud compute forwarding-rules list
NAME                                             REGION    IP_ADDRESS      IP_PROTOCOL  TARGET
a38f1bb8811e511e89c5b42010a8a013                 us-west1  35.197.65.23    TCP          us-west1/targetPools/a38f1bb8811e511e89c5b42010a8a013
a5f61b2bd126f11e89c5b42010a8a013                 us-west1  35.203.177.233  TCP          us-west1/targetPools/a5f61b2bd126f11e89c5b42010a8a013
ac78780fb11e111e8858442010a8a020                 us-west1  35.230.48.211   TCP          us-west1/targetPools/ac78780fb11e111e8858442010a8a020
k8s-fw-fortio-fortio-ingress--71cb3990c3cf10a6             35.227.193.162  TCP          k8s-tp-fortio-fortio-ingress--71cb3990c3cf10a6
k8s-fw-fortio-fortio-ingress2--6c90c323237f439e            35.190.69.114   TCP          k8s-tp-fortio-fortio-ingress2--6c90c323237f439e

I would expect the gce ingress controller to remove these resources when the ingress or cluster is deleted.

[nicksardo]

This is a known issue and being worked on.

[danehans]

@nicksardo Can you share a link to the known issue? I searched through the repo issue list before opening this issue and could not find a match.

[nicksardo]

@danehans The issue is internal to Google. When you delete the cluster, GKE does not wait for the ingress controller to clean up resources. If you want to avoid orphaning resources, delete all the ingress objects and then wait for all the related GCP resources to be deleted prior to calling gcloud container clusters delete...

You can delete the orphan resources through the UI or via gcloud.
Within GCP Console, go to Network Services->Load Balancing and delete the orphaned load balancers. Click on the "advanced" link to see if any resources remain (like backend services) and delete them too.
If you want to use gcloud, start at the front of the load balancer resources and work backwards (the opposite order you have listed)

[danehans]

I have been manually deleting the resources using gcloud. For others that may come across this issue:

$ gcloud compute forwarding-rules delete $ROUTE_RULE_NAME --global -q
$ gcloud compute target-http-proxies delete $TARGET_HTTP_PROXIES -q
$ gcloud compute url-maps delete $URL_MAPS -q
$ gcloud compute backend-services delete --global -q $BACKEND_SERVICES
$ gcloud compute instance-groups unmanaged delete -q $K8S_INSTANCE_GROUPS

[ahmetb]

@danehans there are a few more to delete:

gcloud compute health-checks delete -q [NAME]
gcloud compute firewall-rules delete -q [NAME]

Also the static IPs that are not used are actually billed, so if you want to release that:

gcloud compute addresses delete --global -q [NAME]

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nicksardo]

/remove-lifecycle stale

[tx-ares]

Any update on this?

[bowei]

GKE currently gives a grace period to allow the controller to delete the resources, so the resources should get a chance to be deleted. It may be that the controller is taking longer than the grace period to do the GC, which we are working on.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale