GCE ingress stucks on "Creating ingress" status, existing ingresses don't update

[wtaras]

GCE Ingress stucks on "Creating ingress" status.
Also changes on existing ingresses don't appear on load balancer.

Cluster version: 1.10.7-gke.1

Ingress is creating without any custom properties:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: dev-namespace
  creationTimestamp: 2018-09-11T08:53:30Z
  generation: 1
  name: main
  namespace: dev
  resourceVersion: "36830410"
  selfLink: /apis/extensions/v1beta1/namespaces/dev/ingresses/main
  uid: 282ec501-b5a0-11e8-97b0-42010a8002e3
spec:
  rules:
  - host: backend1.domain
      paths:
      - backend:
          serviceName: backend1
          servicePort: 3001
  - host: backend2.domain
    http:
      paths:
      - backend:
          serviceName: backend2
          servicePort: 8080
  - host: backend3.domain
    http:
      paths:
      - backend:
          serviceName: backend3
          servicePort: 80
status:
  loadBalancer: {}

Name:             main
Namespace:        dev
Address:
Default backend:  default-http-backend:80 (10.52.2.12:8080)
Rules:
  Host                                  Path  Backends
  ----                                  ----  --------
  backend1.domain
                                           backend1:3001 (<none>)
  backend2.domain
                                           backend2:8080 (<none>)
  backend3.domain
                                           backend3:80 (<none>)
Annotations:
Events:
  Type    Reason  Age   From                     Message
  ----    ------  ----  ----                     -------
  Normal  ADD     58m   loadbalancer-controller  dev/main
  Normal  ADD     52m   loadbalancer-controller  dev/main
  Normal  ADD     46m   loadbalancer-controller  dev/main
  Normal  ADD     41m   loadbalancer-controller  dev/main
  Normal  ADD     35m   loadbalancer-controller  dev/main
  Normal  ADD     30m   loadbalancer-controller  dev/main
  Normal  ADD     24m   loadbalancer-controller  dev/main
  Normal  ADD     18m   loadbalancer-controller  dev/main
  Normal  ADD     13m   loadbalancer-controller  dev/main
  Normal  ADD     7m    loadbalancer-controller  dev/main
  Normal  ADD     59s   loadbalancer-controller  dev/main

I don't see any error/warn messages on cluster/balancer logs.
How can I find problem cause and fix it?

[rramkumar1]

@taras-stotsky Are you on GCE or GKE?

If you are on GKE, email me your cluster location, project name and cluster name and I can take a look for you.

[wtaras]

@rramkumar1 yes, I'm on GKE. Sent email now, thanks.

[rramkumar1]

@taras-stotsky See #471

[rramkumar1]

@taras-stotsky Please post here if the workaround detailed in the linked issue above works for you.

[wtaras]

Yes, now workaround works, ingress creates correctly without stuck after recreating.
But before @rramkumar1 took a look on our cluster recreating didn't help.

[wtaras]

@rramkumar1 sorry, workaround worked for me only with 1 single ingress. When I tried recreate another ingress in same cluster - stuck with "Creating ingress" again. How can I use workaround without worrying you?

[rramkumar1]

@taras-stotsky The bulletproof approach would be to just recreate all your ingresses. What's happening is that one or more of your ingresses is running into the issue described in the above issue and that's blocking the sync of other ingresses.

This is unfortunate, but we should have a fix coming in the next week or so.

[wtaras]

that's blocking the sync of other ingresses.

Ok, now I realize. Thank you again!

[Miyurz]

This is happening with GKE ingress controller now. I am trying to create an internal ingress and its stuck in "creating state"

apiVersion: extensions/v1beta1    
kind: Ingress    
metadata:    
  name: resource-services-ingress    
  namespace: staging    
  annotations:    
    kubernetes.io/ingress.class: "gce-internal"    
    kubernetes.io/ingress.allow-http: "false"

[elavenrac]

Can confirm I am also running into this issue as of today.

[Miyurz]

Can confirm I am also running into this issue as of today.

I upgraded k8s master version to 1.16+, deleted the ingress, and reapplied which fixed the issue for me.

[elavenrac]

Thanks @Miyurz! We upgraded our cluster (nodes + master) to 1.16+ and the Internal LB Ingress deployed successfully.