On this page
In the first two articles you learned that a Service claims every Pod whose labels match its selector, and forwards traffic to them. That rule has an uncomfortable consequence: the moment a Pod exists with the right labels, the Service starts sending it real user requests.
Nobody asked the application whether it was ready. A container can be running while its process is still loading configuration, warming a cache, or waiting on a database. During those seconds, a share of your users get errors, and nothing in the cluster looks wrong.
Probes close that gap. They are configured on the Pod, not on the Service, but they are what decides whether a Pod is allowed to appear behind a Service at all. This is the third and final article on Services.
We will cover the following topics:
- The three probes and the distinct question each one answers
- The four probe mechanisms: HTTP, command, TCP, and gRPC
- Every timing parameter, and what happens when you get them wrong
- Watching a Pod leave and rejoin a Service without ever restarting
- Watching a Pod get restarted by a failing liveness probe
Technical Requirements
- A working Kubernetes cluster and
kubectl. - Docker and the .NET 10 SDK if you want to build the sample image.
Core Concepts
Three Probes, Three Questions
The names sound similar and the YAML looks almost identical, which is why they get confused. The way to keep them straight is to remember that each answers a different question and each has a different consequence when it fails.
| Probe | Question | On failure |
|---|---|---|
| startupProbe | Has the application finished starting? | The container is restarted, but only after the whole allowance is used up. |
| readinessProbe | Can this Pod serve requests right now? | The Pod is removed from Service endpoints. It is not restarted. |
| livenessProbe | Is this container broken beyond recovery? | The container is killed and restarted. |
The distinction that matters most is between the last two. Readiness is reversible and gentle: a Pod that fails it stops receiving traffic but keeps running, and it rejoins the moment it recovers. Liveness is a death sentence: failing it restarts the container and throws away whatever state it held.
Get this backwards and you cause outages. A liveness probe that checks a database connection will restart every one of your Pods when the database has a hiccup, turning a brief dependency blip into a full restart storm. That check belongs in the readiness probe, where the Pods quietly leave the load balancer and come back when the database returns.
One more thing about liveness: it cannot repair anything. It only detects that a container is broken and orders its termination, hoping a fresh start behaves better.
How startupProbe Protects the Other Two
Legacy applications often need a long time on first start. That creates a dilemma: liveness probes should react fast to catch deadlocks, but a fast liveness probe will kill a slow starter before it ever finishes booting.
You can paper over it with a large initialDelaySeconds, but that delay applies to every check cycle decision, so you lose fast detection forever in exchange for a slow start once.
startupProbe is the proper answer. While it is running, the readiness and liveness probes are suspended entirely. Give it a generous failureThreshold * periodSeconds budget, and the application gets all the time it needs. The instant it succeeds, the startup probe never runs again and the other two take over with tight, fast settings.
The Four Probe Mechanisms
All three probes support the same set of mechanisms. Pick the one that matches your workload.
| Mechanism | Succeeds when | Good for |
|---|---|---|
httpGet | The response code is at least 200 and below 400 | Web services and APIs. The most common choice. |
exec | The command exits with code 0 | Anything with a CLI health check, such as a database client. |
tcpSocket | A TCP connection can be opened | Non-HTTP protocols such as LDAP or a message broker. |
grpc | The gRPC Health Checking Protocol reports serving | gRPC services that implement the standard health service. |
The HTTP range catches people out. A 404 is a failure, so probing a path that does not exist means the Pod is never healthy. That is a real and common mistake: pointing a probe at /healthcheck on a stock nginx image gives a 404 forever.
HTTP probes also accept host, scheme, path, httpHeaders, and port. A custom header is worth adding, because it lets you filter probe traffic out of your access logs.
What Makes a Good Readiness Endpoint
A readiness endpoint that returns 200 unconditionally is decoration. It proves the web server is listening and nothing else.
Make it actually test what the application needs to serve a request: open a connection to the database, check that a required cache is reachable, verify that configuration finished loading. If you write the application, add a dedicated endpoint that touches every backend and reports the result. That is the endpoint worth probing.
Keep it cheap, though. It runs every few seconds on every replica forever, so it should not do heavy work or cascade into other services.
The Timing Parameters
All three probes share the same tuning fields.
| Field | Meaning | Default |
|---|---|---|
initialDelaySeconds | Wait before the first check | 0 |
periodSeconds | Interval between checks | 10 |
timeoutSeconds | How long a single check may take before counting as a failure | 1 |
successThreshold | Consecutive successes needed to be considered healthy again | 1 |
failureThreshold | Consecutive failures needed to act | 3 |
The default timeoutSeconds: 1 is the field that most often causes mysterious flapping. A busy application that occasionally takes 1.2 seconds to answer will be marked unhealthy for no good reason. Raise it.
Note that successThreshold must be 1 for liveness and startup probes; Kubernetes rejects any other value. Only readiness allows a higher number.
The time before a probe acts is failureThreshold × periodSeconds, plus the initial delay. With the defaults, a dead container takes about thirty seconds to be restarted. Decide deliberately whether that is too slow or too aggressive for your workload.
Hands-On: Kubernetes Commands
See which Pods are actually receiving traffic
The ready condition on each endpoint is what the Service uses.
Check readiness and restart count together
These two columns separate a readiness problem from a liveness problem at a glance.
Read the probe configuration on a running Pod
Confirms which paths, ports, and thresholds are actually in effect.
Find out why a Pod is being restarted
The kubelet records both the failure and its decision.
Watch readiness change in real time
The READY column flips without the restart count moving.
Step-by-Step Example
The Scenario
A wallet API with three replicas behind a ClusterIP Service. The application deliberately takes twenty seconds to start, and exposes two administrative switches that let us break readiness or liveness on demand. That turns two behaviours that are normally hard to observe into experiments you can run in a minute.
Step 1 — Build the Application
Replace Program.cs. Notice that each probe endpoint answers a different question, matching the table above:
Add a Dockerfile and build:
Step 2 — Deploy With All Three Probes
Read the three probe blocks together. The startup probe is generous, the readiness probe is quick to react, and the liveness probe is deliberately slower so a brief hiccup does not cause a restart.
Add the Service and a toolbox Pod for testing from inside the cluster:
Step 3 — Watch the Startup Probe Hold Traffic Back
Watch the Pods come up:
Sampled every five seconds, the picture is unambiguous:
For half a minute every Pod is Running but 0/1. The containers started immediately; the applications had not finished initialising, so the startup probe kept failing and the Pods stayed out of the Service.
This is exactly the window in which, without probes, your users would have received errors. Note also that nothing was restarted: the liveness probe was suspended for the whole period, which is the entire point of having a startup probe.
Step 4 — The Key Experiment: Readiness Removes Traffic Without Restarting
Confirm all three Pods are receiving traffic:
Now break readiness on one of them, addressing the Pod directly so we bypass the Service:
Wait about fifteen seconds, then look at the Pod:
Read that line carefully. READY is false, STATUS is still Running, and RESTARTS is still 0. The container was not touched. And the endpoint changed:
Send fifteen requests through the Service and count who answers:
Only two Pods answered. The third is alive and healthy from the kubelet's point of view, but the Service is not sending it anything. Turn readiness back on:
Within a few seconds it rejoins:
That round trip, out of rotation and back, with no restart and no lost state, is what makes readiness the right place for dependency checks.
Step 5 — Liveness Restarts the Container
Now break liveness on a different Pod and watch the opposite happen:
Sample the Pod every five seconds:
Nothing happened for twenty-five seconds, then the restart count went to 1. That delay is failureThreshold: 3 multiplied by periodSeconds: 10. The kubelet waited for three consecutive failures before acting, exactly as configured.
The events say so explicitly:
Compare this with Step 4. Same application, same kind of probe, same HTTP mechanism. The only difference is which probe was failing, and the outcomes could hardly be further apart: one Pod quietly left the load balancer, the other had its container killed.
Note also that the restarted container starts from zero, which means it goes through the twenty-second startup delay again before serving traffic. A liveness restart is not free.
Step 6 — Clean Up
Common Mistakes
Four errors account for most probe-related incidents.
- Checking dependencies in the liveness probe. When the database blips, every replica restarts at once and a small problem becomes an outage. Dependency checks belong in readiness.
- Probing a path that returns 404. Anything outside 200-399 is a failure, so a missing endpoint means the Pod is never healthy. Verify the path exists in the image you are actually running.
- Leaving
timeoutSecondsat its default of 1. A Pod that occasionally answers in 1.2 seconds will flap for no reason. - Using a long
initialDelaySecondsinstead of a startup probe. You pay that delay in slow failure detection forever, rather than once at startup.
Summary
- Probes are configured on the Pod, but they decide which Pods a Service will use.
- readinessProbe removes a Pod from Service endpoints without restarting it. Reversible and gentle.
- livenessProbe kills and restarts the container. It cannot repair anything, only detect and terminate.
- startupProbe suspends the other two until the application has finished initialising, so a slow start is never mistaken for a crash.
- Four mechanisms:
httpGet,exec,tcpSocket, andgrpc. HTTP succeeds on 200-399 only. - Time to act is
failureThreshold × periodSeconds. With the defaults, about thirty seconds. successThresholdmust be 1 for liveness and startup probes.- A good readiness endpoint tests real dependencies, and stays cheap enough to run every few seconds forever.
- Put dependency checks in readiness, never in liveness.
That completes this series on Services. You can now expose Pods inside the cluster, expose them to the outside world, and control precisely when each Pod is allowed to receive traffic.