Multiple Healthcheck Requests from GCP L7

[mattdodge]

TL;DR: For the healthchecks created by the L7 load balancer I am seeing 6 requests, all at once, from each node in my cluster. Would expect to see 1 request per node.

I have an ingress set up to use the GCE ingress controller. It targets multiple services but each behaves the same so I will just use one example. The service's pod is configured with an http readiness probe configured to do a health check every 10 seconds.

readinessProbe:
  httpGet:
    path: /healthcheck
    port: 5000
  timeoutSeconds: 5
  periodSeconds: 10

The GCP Health Check (https://console.cloud.google.com/compute/healthChecks) gets created (although with an interval of 70 seconds, not sure how to make that match the ingress config).

Every 10 seconds I see the kubelet of the node the pod is running on make a health check request to the pod. This is working as expected.

However, every 70 seconds I am seeing six (6) health check requests come from every other node in the cluster. So a cluster with 3 nodes is sending 18 health check requests every 70 seconds.

I have attached a log showing the requests come in if that's helpful. [health_check.log] You can see at the 16:30:57 mark I add 3 more nodes to the cluster resulting in 18 more requests coming through (nodes 10.138.0.[5,6,7]). Also note, the ingress period for this log is actually 25 seconds and GCP health check period is 35 seconds, I was tweaking values during testing.

Main question: how can I get the health checks from GCP to only send one health check every interval? Or maybe, what is the reason for sending 6?
Secondary question: can I configure the GCP health check interval from the ingress configuration?

[nicksardo]

Health check frequency

To ensure high availability, GCP creates redundant copies of each health checker. These redundant health checkers also probe your instances. If any health checker fails, a redundant one can take over with no delay.

If you examine the logs on your instance, you might see health check polling happening more frequently than than you have configured. This is because the redundant health checkers are also following your settings. These redundant health checkers are created automatically and are not separately user configurable.

https://cloud.google.com/compute/docs/load-balancing/health-checks

[mattdodge]

Geez, I pored over docs for so long looking for this. I think I was focusing too much on ingress health check docs. Thanks for sharing, I'll close this since it doesn't really seem to be an issue, just a very redundant feature.