Managing StatefulSets — Network Identity, Scaling, and Deletion

In the earlier articles of this chapter you built a StatefulSet and learned that Kubernetes manages Pod identity and storage, but never your data. This article covers the day-to-day operations: how the headless Service actually produces stable network names, what happens to your data when Pods are destroyed, how scaling behaves differently from a Deployment, and how to delete a StatefulSet without losing or leaking storage.

Everything here is demonstrated with experiments you can repeat. The most important one takes about thirty seconds: we create data in a Pod, delete that Pod, and reconnect using the exact same name. The Pod's IP address changes. The name and the data do not.

We will cover the following topics:

1. How a headless Service produces DNS A records and per-Pod names
2. Why load balancing becomes the client's job
3. Two real use cases: clustered databases and data sharding
4. State persistence when Pods are deleted
5. Ordered scaling, and how to relax it with podManagementPolicy
6. Deleting a StatefulSet with and without its Pods
7. Controlling PVC cleanup with persistentVolumeClaimRetentionPolicy

Technical Requirements

1. A running Kubernetes cluster with a default StorageClass. Docker Desktop, minikube, or a cloud cluster will all work. A multi-node cluster shows the concepts best.
2. kubectl installed and configured.
3. Docker and the .NET 10 SDK if you want to build the sample router image yourself.

Core Concepts

What a Headless Service Really Publishes

A normal Service gets a ClusterIP: a single virtual IP address. Traffic sent to that IP is load balanced across the backing Pods. On Linux this happens at the kernel level, through iptables or IPVS rules that kube-proxy maintains. The client sees one address and never knows which Pod answered.

A headless Service sets clusterIP: None, so there is no virtual IP and no kube-proxy involvement at all. Instead, DNS returns the Pod IP addresses directly. Two kinds of record are published:

1. One A record per ready Pod, all sharing the Service's own name. A lookup of metrics-store-headless returns three addresses when three Pods are ready.
2. One per-Pod name in the form <podName>.<headlessServiceName>.<namespace>.svc.cluster.local. This is the record that gives each replica its stable identity.

The consequence is important: with a headless Service, choosing a target Pod becomes the client's responsibility. Kubernetes hands you a list of addresses and steps back. Some clients take the first address and stay there; others rotate. Nothing balances the load for you.

Why Stable Names Matter

A beginner reasonably asks: why go to all this trouble? Two concrete situations make it necessary.

Clustered databases. Systems such as etcd, MongoDB, or Cassandra need each member to be told the network addresses of the other members. Many of them have no automatic discovery at all. Stable DNS names let you write that member list once and have it stay correct across restarts. There is still a problem when replicas are added or removed during scaling, and this is often solved with a sidecar container that watches the Kubernetes API and rewrites the database configuration as membership changes.

Data sharding. If you split data across replicas, you need a mapping from a logical shard to a physical Pod. Stable DNS names are the natural key for that mapping. Shard 2 always means metrics-store-2, whether that Pod was just rescheduled onto a different node or has been running for a month. We will build exactly this in the hands-on section.

State Persistence

When a StatefulSet Pod is deleted, the controller creates a replacement with the same ordinal name, and that replacement re-attaches the same PersistentVolumeClaim. The container is disposable; the volume is not.

This is true whether the Pod died on its own, was evicted, or you deleted it by hand. It also holds when you delete every Pod at once: Kubernetes recreates them in ascending order, each with its original claim.

The flip side is that the StatefulSet controller will never remove a PersistentVolume as part of deleting a Pod or the StatefulSet itself. Cleaning up storage stays your job, unless you configure the retention policy described later in this article.

Ordered Scaling

Scaling a StatefulSet uses the same tools as a Deployment: edit replicas in the manifest, or run kubectl scale. New Pods are automatically added to the Service's endpoints, and removed when scaled down. The ordering, however, is strict.

1. Deploying N replicas creates Pods sequentially, from ordinal 0 to N-1.
2. Scaling up creates the new Pods sequentially too.
3. Scaling down terminates Pods in reverse order, from N-1 down to 0.
4. Before the next Pod is created, every predecessor must be Running and Ready.
5. Before the next Pod is terminated, every successor must be completely terminated and deleted.

There is a consequence worth thinking about. Because every scaling operation waits for the predecessors to be ready, a single unhealthy Pod blocks the whole queue. If metrics-store-0 fails while you are scaling down from four replicas to one, no further scaling happens to any of the other Pods. Work resumes only when metrics-store-0 becomes ready again.

You can relax this with the .spec.podManagementPolicy field. The default is OrderedReady. Setting it to Parallel makes creation and deletion happen all at once, like a Deployment. Note two limits: this affects only scaling, not rolling updates, and the field is immutable, so you must decide when you create the StatefulSet.

One more scaling note. If you manage a StatefulSet with a Horizontal Pod Autoscaler or a similar tool, leave .spec.replicas out of your manifest entirely. If you set it, every re-apply of the manifest fights the autoscaler and resets the replica count.

Deleting a StatefulSet

There are two ways to delete a StatefulSet, and they differ in what happens to the Pods.

1. Cascading delete (the default) removes the StatefulSet and its Pods. The Pods are terminated first, then the object disappears. Important: this is not the same as scaling to zero and then deleting. In a cascading delete there is no guarantee about termination order, and in practice the Pods usually go down all at once. For a graceful, ordered shutdown, scale to zero first.
2. Orphan delete, with --cascade=orphan, removes only the StatefulSet object. Every Pod keeps running, unmanaged. This is useful during maintenance when you want to replace the controller without disturbing the workload.

In both cases the PVCs and PVs created from volumeClaimTemplates survive by default, so state data is never lost by accident.

Since Kubernetes v1.27 you can change that default with .spec.persistentVolumeClaimRetentionPolicy, which has two independent settings:

whenDeleted controls what happens when the StatefulSet itself is deleted; whenScaled controls what happens to the claims of Pods removed by scaling down. Each accepts Retain (the default) or Delete. The combination above is a common, sensible choice: automatically reclaim storage from replicas you scaled away, but never lose everything because someone deleted the wrong object.

Hands-On: Kubernetes Commands

Resolve the headless Service

Returns one address per ready Pod. The getent tool is available in most Linux images.

Resolve a single Pod by its stable name

Returns exactly one address, no matter how many replicas exist.

Scale a StatefulSet

Works imperatively, or by editing replicas in the manifest and re-applying.

Read the controller's events

Shows each Pod and each claim being created, in order, with timestamps.

See which claim each Pod holds

Filtering the describe output makes the Pod-to-claim mapping easy to read at a glance.

Delete the StatefulSet but keep the Pods

The Pods keep running with no controller watching them.

Change the retention policy on a live StatefulSet

Unlike podManagementPolicy, this field can be changed after creation.

Clean up leftover claims

The StatefulSet controller copies the selector labels onto every claim it creates, so a label selector deletes them all without typing each name.

Step-by-Step Example

The Scenario

We will run a Metrics system in its own namespace. A PostgreSQL StatefulSet named metrics-store holds sensor readings, split across three shards. A metrics-shell Pod gives us a command line for experiments. An ASP.NET Core service named shard-router demonstrates the sharding use case: it maps each sensor name to a shard number, then talks to that shard through its stable per-Pod DNS name.

Step 1 — Create the Namespace, Secret, and Schema

The namespace keeps the exercise isolated and makes cleanup a single command.

The credentials decode to metricsadmin and M3tr1csP@ss2026.

This ConfigMap is mounted into the database image's initialization directory, so every replica creates the same empty readings table when it first starts.

Step 2 — Create the Headless Service

clusterIP: None is what makes this Service headless. It must exist before the StatefulSet, because .spec.serviceName points at it.

Step 3 — Create the StatefulSet

Two fields here are new compared with the previous articles. podManagementPolicy is set to its default OrderedReady explicitly, so you can see it and compare it with the Parallel example later. persistentVolumeClaimRetentionPolicy is also written out with its defaults, so the behaviour is documented in the manifest rather than implied.

Step 4 — Create the Shell Pod

We reuse the postgres:17 image as a client and keep it alive with sleep infinity. The PG* variables are read automatically by psql, so credentials never appear in our commands.

Step 5 — Explore the DNS Records

Ask DNS for the headless Service name. You get one address per ready Pod:

Three A records, pointing straight at Pod IPs. Compare this with a ClusterIP Service, which would return a single virtual address. Here there is nothing to load balance through, which is exactly why the client must choose.

Now ask for each Pod by its own name:

Each name resolves to exactly one Pod. Note that the addresses match the three from the Service lookup, just individually addressable now.

If you prefer the classic nslookup output, run it from a busybox Pod. Use the fully qualified name, because the nslookup in recent busybox images does not apply the cluster's DNS search list reliably:

Step 6 — The Key Experiment: The Name Outlives the Pod

First, note the current IP address of metrics-store-0:

Now write some data through the stable DNS name, and create a database so we have something obvious to look for later:

Delete the Pod and wait for its replacement:

Check the IP address again:

The address changed. Now resolve the name:

The same name now points at the new address, automatically. Any client that stored the name rather than the IP keeps working. Finally, confirm the data came back with the volume:

Both the row and the ststest database survived. The Pod was destroyed and rebuilt, but its identity, its name, and its storage all came back together.

Step 7 — Delete Every Pod at Once

The same guarantees hold when you remove the whole set:

Kubernetes recreates them in ascending order. Check the Pod-to-claim mapping afterwards:

Every Pod found its original claim, and the data is still there:

Step 8 — Build the Shard Router

Now let's use those stable names for something real. The router maps a sensor name to a shard number, then connects to that shard by its per-Pod DNS name. Create the project:

Replace Program.cs with the following:

Read the comment above ShardFor carefully. In .NET, String.GetHashCode() returns a different value in every process, by design. Using it to pick a shard would send the same sensor to a different Pod after every restart, and your reads would silently miss the data. Sharding needs a hash you control.

Add a Dockerfile next to the project file:

On Docker Desktop the cluster shares the local image store, so the image is usable at once. On minikube run minikube image load shard-router:1.0. On a cloud cluster, push to a registry and update the image name below.

The Deployment passes the DNS pattern as an environment variable. The {0} placeholder is replaced with the shard number, producing metrics-store-0.metrics-store-headless and so on.

Step 9 — Watch Sharding in Action

Forward a port and write readings for four different sensors:

Each sensor was routed to a specific Pod by name. Ask the router to describe all shards:

Shard 0 holds one reading: the boiler-temp row we inserted by hand in Step 6, straight into metrics-store-0. The router's hash happens to place the sensor boiler-temp on shard 1, which is why the two do not line up. That is a useful reminder that the shard mapping lives in your application, not in Kubernetes.

Now the real test. Note the address of the Pod holding shard 2, then destroy it:

Query the same sensor again, with no configuration change and no restart of the router:

The Pod's IP address changed, and the reading was still found, down to the original timestamp. This is the sharding use case working exactly as promised: the mapping from logical shard to physical Pod stays correct across a restart because it is expressed as a name, not an address.

Step 10 — Ordered Scaling Up

Scale from three replicas to five and watch closely:

Sampling the Pod list every three seconds makes the ordering unmistakable:

Look at the third line. metrics-store-4 does not even exist until metrics-store-3 reports 1/1. A Deployment would have created both immediately.

The controller's events tell the same story, and show that a claim is created before each new Pod:

Step 11 — Ordered Scaling Down

Now scale back to two replicas:

The highest ordinal goes first. Termination walks backwards down to the target count, and metrics-store-0 is never touched.

Check the claims:

All five survive, because our manifest sets whenScaled: Retain. Two Pods are running and five volumes are being paid for. This is the expected default, and it is why storage cleanup deserves your attention.

Step 12 — Automatic Claim Cleanup on Scale-Down

Change the retention policy so that scaling down also removes the claims. Unlike podManagementPolicy, this field can be patched on a live StatefulSet:

Scale from two replicas down to one and check the claims again:

metrics-data-metrics-store-1 is gone, deleted automatically with its Pod. The claims for ordinals 2, 3, and 4 remain, because they were already orphaned back when the policy still said Retain. Changing the policy does not reach backwards and clean up history.

Choose this setting deliberately. Delete keeps your storage bill honest, but a scale-down then destroys data permanently and immediately.

Step 13 — Deleting Without Touching the Pods

Scale back to three replicas, then delete the controller only:

The StatefulSet is gone, but every Pod is untouched. Look at the ages:

Re-create the StatefulSet from the same manifest. It adopts the running Pods rather than replacing them:

The StatefulSet is five seconds old while the Pods are minutes old. Nothing restarted. Adoption works because the selector labels still match.

Step 14 — Cascading Delete and Manual Cleanup

Now the ordinary delete, which takes the Pods with it:

Remember that this gives no ordering guarantee. The Pods generally go down together. When ordered shutdown matters, scale to zero first and only then delete.

The claims, however, are still there, because our manifest sets whenDeleted: Retain:

Deleting them is a deliberate, manual act. The label selector works because the controller copies the StatefulSet's selector labels onto every claim it creates:

Deleting a claim also releases its PersistentVolume. With a Delete reclaim policy the volume disappears; with Retain it stays behind in a Released state for you to remove.

One warning before you run this. If you plan to work through the next article on rolling out new versions, keep these claims. Deleting them now destroys the database files you would use to verify that state survives an upgrade.

Step 15 — Optional: Parallel Pod Management

To see how different Parallel feels, apply this second, independent StatefulSet. It runs Redis, needs no storage, and exists purely for the contrast. The file contains both the headless Service and the StatefulSet.

All four Pods exist three seconds in, and they become ready in whatever order they finish. Compare this with Step 10, where the fourth Pod had not been created yet after fifteen seconds. Note that the Pods still get ordered names and their own claims. Only the timing changed.

Step 16 — Clean Up

Deleting the namespace removes everything in one command.

Confirm no volumes were left behind.

Summary

In this article you learned how to operate a StatefulSet day to day:

1. A headless Service publishes one A record per ready Pod plus a per-Pod name of the form <pod>.<service>.<namespace>.svc.cluster.local.
2. There is no kube-proxy load balancing behind a headless Service. The client chooses the target.
3. A Pod's IP address changes when it is recreated. Its DNS name and its volume do not.
4. Stable names make clustered databases and data sharding practical, because a shard-to-Pod mapping stays valid across restarts.
5. Scaling is ordered: up from 0, down in reverse, each step waiting for the previous Pod. A single unhealthy Pod stalls the queue.
6. podManagementPolicy: Parallel removes that ordering for scaling only. It is immutable after creation.
7. Leave replicas unset when a Horizontal Pod Autoscaler manages the StatefulSet.
8. --cascade=orphan deletes the controller and leaves the Pods running, and re-applying the manifest adopts them.
9. A cascading delete gives no termination order guarantee. Scale to zero first for a graceful shutdown.
10. persistentVolumeClaimRetentionPolicy controls claim cleanup separately for whenDeleted and whenScaled. Both default to Retain.

In the next article we will release a new version of an application running as a StatefulSet, and look at how revisions and update strategies control those ordered rollouts.