Controller not syncing LoadBalancer IP when certificate is invalid

[ypal]

Running on GKE 1.12.6-gke.10

When creating an Ingress with the following manifest:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: qwerty-com
  annotations:
    kubernetes.io/ingress.class: "gce"
    certmanager.k8s.io/acme-http01-edit-in-place:  "true"
    certmanager.k8s.io/issuer: letsencrypt-production-htpp01
    kubernetes.io/ingress.allow-http: "true"
spec:
  tls:
  - secretName: querty-com-tls
    hosts:
    - qwerty.com
  rules:
  - host: qwerty.com
    http:
      paths:
      - path: "/*"
        backend:
          serviceName: echoheaders
          servicePort: 80

When I run kubectl describe ingress qwerty-com I get the following warning:

Events:
  Type     Reason             Age                  From                     Message
  ----     ------             ----                 ----                     -------
  Normal   ADD                5m31s                loadbalancer-controller  yaco/qwerty-com
  Normal   CreateCertificate  5m31s                cert-manager             Successfully created Certificate "querty-com-tls"
  Warning  Sync               13s (x12 over 5m8s)  loadbalancer-controller  Error during sync: error running load balancer syncing routine: loadbalancer yaco-qwerty-com--32ca57de52134e26 does not exist: Cert creation failures - k8s-ssl-1b9d6041f1c8eb03-e3b0c44298fc1c14--32ca57de52134e26 Error:googleapi: Error 400: Invalid value for field 'resource.certificate': ''. A certificate must be specified for SSL certificate creation., invalid

The Ingress Resource never get's the IP in it's status even though the LoadBalancer was successfully created, the controller updates the host/path rules when you update the ingress manifest.

$ kubectl get ing qwerty-com -oyaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    certmanager.k8s.io/acme-http01-edit-in-place: "true"
    certmanager.k8s.io/issuer: letsencrypt-production-htpp01
    kubernetes.io/ingress.allow-http: "true"
    kubernetes.io/ingress.class: gce
  creationTimestamp: "2019-04-18T10:26:57Z"
  generation: 3
  name: qwerty-com
  namespace: yaco
  resourceVersion: "152961905"
  selfLink: /apis/extensions/v1beta1/namespaces/yaco/ingresses/qwerty-com
  uid: 7eae7511-61c4-11e9-8d95-42010a840ff6
spec:
  rules:
  - host: qwerty.com
    http:
      paths:
      - backend:
          serviceName: echoheaders
          servicePort: 80
        path: /blabla
      - backend:
          serviceName: echoheaders
          servicePort: 80
        path: /*
      - backend:
          serviceName: cm-acme-http-solver-2kg64
          servicePort: 8089
        path: /.well-known/acme-challenge/5Wo3u_5jU_Gr9KaAQfiKCTNAWULQvBO7OazQRe4AXVA
  tls:
  - hosts:
    - qwerty.com
    secretName: querty-com-tls
status:
  loadBalancer: {}

I am planning on using external-dns to link the LoadBalancer IP to a domain but this is preventing this. If I remove all TLS related configuration or if I put a valid certificate it works perfectly fine, but having to to this for new domains makes automating the whole process much harder.

[rramkumar1]

@ypal Why are you creating an Ingress spec with the "tls" field specified if you are using cert manager? Isn't cert manager supposed to populate that field for you?

[rramkumar1]

@ypal Nevermind, I just remembered that cert manager creates the secret for you but you still have to add it to the Ingress spec beforehand.

Is it possible that when you actually created the secret and the tls.cert key was populated with an empty string. Would it be possible to post here the contents of that secret?

[rramkumar1]

ping @ypal

[ypal]

cert-manager changed it behaviour recently, now creates a secret without a tls.crt and it updates it later with the cerrtificate once it's issued.

I found weird that ingress-gce updates the GCLB if you add new paths, but won't update the k8s service with the IP

Example secret:

Name:         qwerty-com-tls
Namespace:    default
Labels:       certmanager.k8s.io/certificate-name=querty-com-tls
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
ca.crt:   0 bytes
tls.crt:  0 bytes
tls.key:  1675 bytes

[rramkumar1]

@ypal Okay, then what you are seeing makes perfect sense given the behavior change in cert manager.

I think it would be good if you file an issue with them and explain the issue. I imagine its going to be very difficult for us to make a change here to alleviate this but I could be wrong.

[ypal]

One of the last versions of cert-manager started issuing temporary self-signed certificates while it issues the right one. Updating cert-manager made this works as a certificate is there.

I find it weird that the syncing derps with a missing certificate, I was expecting it to just warn about it and continue with the rest of the sync.

I'll close this issue, thanks @rramkumar1 😄