Why do I need 3 different kind of probes in kubernetes: startupProbe, readinessProbe, livenessProbe

Question

Why do I need 3 different kind of probes in kubernetes:

• startupProbe
• readinessProbe
• livenessProbe

There are some questions (k8s - livenessProbe vs readinessProbe, Setting up a readiness, liveness or startup probe) and articles about this topic. But this is not so clear:

• Why do I need 3 different kind of probes?
• What are the use cases?
• What are the best practises?

Answer 1

These 3 kind of probes have 3 different use cases. That's why we need 3 kind of probes.

Liveness Probe

If the Liveness Probe fails, the pod will be restarted (read more about failureThreshold).

Use case: Restart pod, if the pod is dead.

Best practices: Only include basic checks in the liveness probe. Never include checks on connections to other services (e.g. database). The check shouldn't take too long to complete. Always specify a light Liveness Probe to make sure that the pod will be restarted, if the pod is really dead.

Startup Probe

Startup Probes check, when the pod is available after startup.

Use case: Send traffic to the pod, as soon as the pod is available after startup. Startup probes might take longer to complete, because they are only called on initializing. They might call a warmup task (but also consider init containers for initialization). After the Startup probe succeeds, the liveliness probe is called.

Best practices: Specify a Startup Probe, if the pod takes a long time to start. The Startup and Liveness Probe can use the same endpoint, but the Startup Probe can have a less strict failure threshhold which prevents a failure on startup (s. Kubernetes in Action).

Readiness Probe

In contrast to Startup Probes Readiness Probes check, if the pod is available during the complete lifecycle. In contrast to Liveness Probes only the traffic to the pod is stopped, if the Readiness probe fails, but there will be no restart.

Use case: Stop sending traffic to the pod, if the pod can not temporarily serve because a connection to another service (e.g. database) fails and the pod will recover later.

Best practices: Include all necessary checks including connections to vital services. Nevertheless the check shouldn't take too long to complete. Always specify a Readiness Probe to make sure that the pod only gets traffic, if the pod can properly handle incoming requests.

Documentation

• This article explains very well the differences between the 3 kind of probes.
• The Official kubernetes documentation gives a good overview about all configuration options.
• Best practises for probes.
• The book Kubernetes in Action gives most detailed insights about the best practises.

Answer 2

Here's a concrete example of one we're using in our app. It has a single crude HTTP healthcheck, accessible on http://hostname:8080/management/health.

ports:
  - containerPort: 8080
    name: web-traffic

App Initialization (startup)

• Spring app that is slow to start - anywhere between 30-120 seconds.
• Don't want other probes to run until app is started.
• Check it every 10 seconds (give up after 5s) for up to 180s before k8s gets into a crash loop.

startupProbe:
  successThreshold: 1
  failureThreshold: 18
  periodSeconds: 10
  timeoutSeconds: 5
  httpGet:
    path: /management/health
    port: web-traffic

Healthcheck (readiness)

• Ping the app every 10 seconds to make sure it's healthy (ie. accepting HTTP requests).
• If fail two subsequent pings, cordone it off (prevents cascades).
• Must pass two subsequent health checks before can accept traffic again.

readinessProbe:
  successThreshold: 2
  failureThreshold: 2
  periodSeconds: 10
  timeoutSeconds: 5
  httpGet:
    path: /management/health
    port: web-traffic

App has Died (liveliness)

• If app fails 3 consecutive health checks, 30 seconds apart, reboot the container. Maybe app got into an unrecoverable state like Java ran out of heap memory.

livenessProbe:
  successThreshold: 1
  failureThreshold: 3
  periodSeconds: 30
  timeoutSeconds: 5
  httpGet:
    path: /management/health
    port: web-traffic

Answer 3

The difference between livenessProbe, readinessProbe, and startupProbe

livenessProbe:

 livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3

• It is used to indicate if the container has started and is alive or not i.e. proof of being avaliable.
• In the given example, if the request fails, it will restart the container.
• If not provided the default state is Success.

readinessProbe:

 readinessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3

• It is used to indicate if the container is ready to serve traffic or not i.e.proof of being ready to use.
• It checks dependencies like database connections or other services your container is depending on to fulfill its work.
• In the given example, until the request returns Success, it won't serve any traffic(by removing the Pod’s IP address from the endpoints of all Services that match the Pod).
• Kubernetes relies on the readiness probes during rolling updates, it keeps the old container up and running until the new service declares that it is ready to take traffic.
• If not provided the default state is Success.

startupProbe:

 startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 3

• It is used to indicate if the application inside the Container has started.
• If a startup probe is provided, all other probes are disabled until the startup probe succeeds.
• In the given example, if the request fails, it will restart the container.
• Once the startup probe has succeeded once, the liveness probe takes over to provide a fast response to container deadlocks.
• If not provided the default state is Success.

Check K8S documenation for more.

Answer 4

I think the below table describes the use-cases for each.

Feature	Readiness Probe	Liveness Probe	Startup Probe
Exmine	Indicates whether the container is ready to service requests.	Indicates whether the container is running.	Indicates whether the application within the container is started.
On Failure	If the readiness probe fails, the endpoints controller removes the pod's IP address from the endpoints of all services that match the pod.	If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy.	If the startup probe fails, the kubelet kills the container, and the container is subjected to its restart policy.
Default Case	The default state of readiness before the initial delay is Failure. If a container does not provide a readiness probe, the default state is Success.	If a container does not provide a liveness probe, the default state is Success.	If a container does not provide a startup probe, the default state is Success.

Sources:

• Kubernetes in Action

Answer 5

This topic very interesting but I would like to add 1 more detail (important from my point of view).

I was not able to understand the difference between startup probe + liveness probe from one side and liveness probe with extended amount of failedTheshold.

And looks like differenceis in a nature of startup probe to be a single time executed. In contrast liveness probe and readiness probe are cyclic (or periodic, repeatable).

If start up probe is passed - it won't be executed until pod is alive. So the idea of startup probe to give a bit more time for application startup.

Lets consider example

readinessProbe:
  initialDelay: 20
  failureThreshold: 2
  periodSeconds: 10

livenessProbe:
  initialDelay: 10
  failureThreshold: 3
  periodSeconds: 5

We wait for 40 sec [(20 + 210)] startup and then 25 sec [(10 + 53)] for become alive but consider application is not alive after 25 [(10 + 5*3)] seconds of unavailability

Answer 6

startupProbe: Purpose: The startup probe determines whether an application within a container has successfully started and is ready to handle requests. LivenessProbe: Purpose: The liveness probe monitors the ongoing health of a container and determines whether it is running properly. Readiness Probe: Purpose: The readiness probe determines whether a container is ready to handle incoming requests and serve traffic.