Data Management in StatefulSets — Replication, Storage, and Backups

In the previous article you learned what a StatefulSet is and how it gives each Pod a sticky name, a stable DNS record, and its own PersistentVolumeClaim. Those guarantees are powerful, but they can be misleading. A StatefulSet manages Pod identity and storage. It does not manage your data.

This is the single most common misunderstanding among students starting with stateful workloads. If you create a StatefulSet with three database replicas, you do not get one highly available database. You get three separate, empty databases that happen to be named in order. Nothing copies rows between them.

In this article we will make that fact visible with a real experiment, and then look at the techniques you need to fill the gap: data initialization, replication, storage accessibility, and backups.

We will cover the following topics:

1. The three data responsibilities that stay with you, not with Kubernetes
2. Replication management techniques: init containers, application-level replication, external scripts
3. The anatomy of a StatefulSet specification, field by field
4. Building a seeded PostgreSQL StatefulSet with an init container
5. Proving that replicas do not share data
6. Taking an external backup and restoring it into another replica

Technical Requirements

1. A running Kubernetes cluster with a default StorageClass. Docker Desktop, minikube, or any cloud cluster will do.
2. kubectl installed and configured.
3. Docker and the .NET 10 SDK if you want to build the sample API image yourself.

Core Concepts

What StatefulSets Do Not Do for You

A StatefulSet is a controller for Pod life cycle and storage binding. Everything about the content of that storage is your responsibility. There are three areas where beginners are usually surprised.

Data cloning and synchronization

Unlike stateless applications, stateful applications rely on persistent data. A StatefulSet manages Pod ordering and identity, but it does not replicate data between Pods. When warehouse-db-1 starts, it gets a brand new empty volume. It does not receive a copy of what warehouse-db-0 holds.

You have to implement this yourself. The common approaches are init containers that copy data from a predefined source during Pod creation, built-in replication features inside your application such as PostgreSQL streaming replication or MySQL replication, and external scripts that manage synchronization.

Remote storage accessibility

StatefulSets can reschedule Pods onto any available node in the cluster. For data to survive that move, the storage behind the PersistentVolume must be reachable from every worker node. A volume tied to one machine's local disk breaks this.

In practice this means choosing a StorageClass backed by network-attached storage such as a cloud disk, NFS, or a CSI driver that replicates across nodes. On a local learning cluster you will typically use a hostPath-style provisioner, which does not follow the Pod between nodes. That is fine for exercises, but never for production.

External backups

StatefulSets handle Pod life cycles and data persistence inside the cluster. They do not protect you from a catastrophic event that takes the cluster or the storage backend with it. A deleted namespace, a corrupted volume, or a bad migration will destroy your data just as thoroughly with a StatefulSet as without one.

Disaster recovery needs a separate backup solution that writes outside the cluster: dumps pushed to cloud object storage, volume snapshots retained by your cloud provider, or a dedicated backup server. We will build a simple version of this in the hands-on section.

Replication Management Techniques

Since Kubernetes will not synchronize your data, you must choose a strategy. The table below compares the three usual options.

For anything beyond an exercise, prefer application-level replication, and prefer a mature operator over hand-written scripts. The official Kubernetes tutorial on running a Cassandra cluster is a good example of how much work joining and removing replicas really involves. The example in this article stays deliberately simple so that the underlying behavior stays visible.

Anatomy of a StatefulSet Specification

The StatefulSet spec looks much like a Deployment, with a few extra fields for storage and networking. There are five main components you must understand.

1. replicas — the number of Pod replicas that should run from the given template. Pods are created or deleted to maintain this number.
2. serviceName — the name of the headless Service that governs the StatefulSet and provides the network identity for its Pods. This Service must exist before the StatefulSet is created.
3. selector — a label selector defining which Pods the StatefulSet owns. It supports both equality-based and set-based selectors, and it must match the labels in the Pod template.
4. template — the Pod template. Pod names are not random; they follow the <statefulSetName>-<ordinal> convention. You can optionally use .spec.ordinals to change the number the first Pod starts from.
5. volumeClaimTemplates — the template for the PersistentVolumeClaim created for each Pod. Every Pod gets its own PVC, permanently associated with that Pod name. PVC names follow the <volumeClaimTemplateName>-<statefulSetName>-<ordinal> convention.

The Default StorageClass

When a volumeClaimTemplates entry does not set storageClassName, the PVCs are provisioned by the cluster's default StorageClass. That class is the one carrying the storageclass.kubernetes.io/is-default-class annotation. Which class this is depends entirely on how your cluster was deployed.

Check the annotation on your own cluster:

On a Docker Desktop cluster the output looks like this:

In minikube the default is a class named standard using the k8s.io/minikube-hostpath provisioner. In Azure Kubernetes Service it is a class named default backed by Azure disks. Because the provisioner differs, the exact output of the commands in this article will differ slightly on your cluster.

Hands-On: Kubernetes Commands

These are the commands used most often when inspecting data and storage in a StatefulSet.

Inspect the StatefulSet and its Pods

sts is the accepted abbreviation for statefulset.

Find which PVC a Pod is using

The Volumes section of the describe output names the claim bound to that specific Pod.

List claims and the volumes behind them

PVCs are namespaced; PersistentVolumes are cluster-wide, so the second command needs no namespace.

Read the init container's log

Use -c to select a container when a Pod has more than one.

Run a command against one specific replica

Because names are deterministic, you can address an exact replica.

Copy a file out of a Pod

Useful for pulling a database dump onto your machine.

Stream a backup straight to a local file

Redirecting the output of kubectl exec avoids writing the dump inside the container at all.

Step-by-Step Example

The Scenario

We will build a Warehouse stock system in its own namespace. A PostgreSQL StatefulSet named warehouse-db runs three replicas. Each replica is seeded at first start with a small stock table by an init container and the database image's own initialization directory.

Alongside it we run two clients: a db-toolbox Pod holding the psql command line client, and an ASP.NET Core API named warehouse-api that reads the stock table and reports which replica answered. That last detail is what makes the lesson of this article visible.

Step 1 — Create the Namespace

Keeping the exercise in its own namespace makes cleanup a single command later.

Step 2 — Create the Secret

The database credentials go in a Secret, base64-encoded. The decoded values are warehouseadmin and Wh0us3P@ss2026.

You can produce the same Secret imperatively, which is handy when you do not want the values in a file at all. Generate base64 values with echo -n 'warehouseadmin' | base64, or skip the encoding entirely and let kubectl do it:

Step 3 — Create the Seed ConfigMap

This ConfigMap holds the SQL that creates and fills the stock table. It is the "predefined source" that our init container will copy from, and it is also mounted into the database container's initialization directory.

Step 4 — Create the Headless Service

The Service must exist before the StatefulSet, because .spec.serviceName refers to it. Note that the selector uses two labels, app and environment, to show that equality-based selectors can combine several keys.

Step 5 — Create the StatefulSet with an Init Container

This manifest is the heart of the article. Three things deserve your attention:

1. The init container seed-copier runs before PostgreSQL starts. It copies the SQL file from the ConfigMap onto the persistent volume. This demonstrates the "copy data from a source into persistent storage" pattern in its simplest possible form.
2. The same ConfigMap is also mounted at /docker-entrypoint-initdb.d. The official PostgreSQL image runs every .sql file it finds there, but only when it initializes an empty data directory. This is application-level initialization.
3. volumeClaimTemplates requests a 1 GiB ReadWriteOnce volume per Pod. No storageClassName is set, so the default StorageClass is used.

Note why PGDATA points to a subdirectory of the mount. PostgreSQL refuses to initialize into a directory that is not empty, and our init container has just written a seed folder at the root of the volume. Putting the database files in pgdata keeps the two apart.

Step 6 — Watch the StatefulSet Come Up

Wait for all three replicas to become ready. They start one at a time, in order.

Confirm the object and its Pods:

Step 7 — Verify the Init Container Ran

The init container has already finished, but its log is still available:

The file it wrote lives on the volume, so the database container can see it too:

And the initialization directory did its job, so the table exists with its three rows:

Step 8 — Inspect the Per-Pod Storage

Look at which claim each Pod received:

The Volumes section shows the claim bound to this Pod and nothing else:

Repeat the command for warehouse-db-1 and you will see warehouse-data-warehouse-db-1 instead. Each Pod has its own claim, following the <volumeClaimTemplateName>-<statefulSetName>-<ordinal> convention.

The PersistentVolumes behind those claims are cluster-scoped, so list them without a namespace:

The RECLAIM POLICY column says Delete here, which means these volumes disappear when their claims are deleted. On a production cluster you would usually want Retain, so an accidental claim deletion does not destroy the data.

Step 9 — Deploy the Toolbox Pod

To query the databases we need a Pod with a PostgreSQL client. Rather than pulling a separate utility image, we reuse the postgres:17 image and keep it alive with sleep infinity. The PGUSER, PGPASSWORD, and PGDATABASE variables are read automatically by psql, so we never have to type credentials.

Step 10 — The Key Experiment: Replicas Do Not Share Data

First, confirm that all three replicas were seeded independently. Each one ran the ConfigMap script against its own empty volume, so each has the same three rows for the same reason, not because anything was copied between them.

A small shell loop asks each replica in turn. The -tAX flags strip the table borders so that psql prints just the number.

Now write a new row to one replica only:

Run the same counting loop again:

This is the whole lesson in one line. The StatefulSet is healthy, all three Pods are ready, and the data has silently diverged. Nothing in Kubernetes will ever reconcile this. If you had expected these three Pods to behave like one database, you would now be serving wrong answers to a third of your users.

Step 11 — Why the Headless Service Makes This Dangerous

Connect through the headless Service name instead of a specific Pod, and ask the server which address answered:

The headless Service resolved to one of the three Pod IPs, and that Pod happens to be a replica without our new row. Which address a client picks is decided by its DNS resolver, not by you. Different clients can settle on different replicas and then stay there.

For read-write traffic, never point an application at the headless Service name of a database StatefulSet. Address a specific Pod, or put a proper primary-election mechanism in front of it.

Step 12 — Confirm the Correct Pattern from an Application

Let's prove the correct approach with a small ASP.NET Core service. It reads the stock table and reports the IP of the server that answered, so we can see exactly which replica it used. Create the project:

Replace Program.cs with the following:

Pooling=false is set deliberately. Connection pooling would hide which replica is being used by reusing one connection, and for this exercise we want every request to resolve the host again.

Add a Dockerfile beside the project file:

On Docker Desktop the cluster shares the local image store, so the image is available immediately. On minikube run minikube image load warehouse-api:1.0. On a cloud cluster, push to a registry and change the image name below.

Now the Deployment. Look closely at WAREHOUSE_DB_HOST: it names the Pod warehouse-db-0, not the Service. That is the whole point of Step 11.

Forward a port and call the endpoint a few times:

The response names the replica that answered, followed by the rows it returned (formatted for readability):

Call it several times and servedByPodIp never changes, because the host name names one Pod:

That address belongs to warehouse-db-0, the replica holding our extra row. Compare it with the toolbox result from Step 11, which reached a different replica and saw only three rows. Two clients, one StatefulSet, two different answers. The client that named a specific Pod is the one behaving predictably.

Step 13 — Take an External Backup

Now let's address the third responsibility: backups. Stream a dump out of the cluster and onto your machine in a single command. Nothing is written inside the container.

The --clean --if-exists flags make the dump drop existing objects before recreating them. Without them, restoring into a database that already has the stock table fails with a duplicate key error.

Check that the dump really contains your data:

A file on your laptop is not a backup strategy, but the shape is right. In production you would run this from a CronJob and push the output to object storage outside the cluster.

Step 14 — Restore into Another Replica

This is the "external scripts" replication technique in its most basic form. We take the dump from warehouse-db-0 and load it into warehouse-db-1, pulling the two replicas back into agreement.

The -i flag is required so that the local file is piped into the Pod's standard input. Now count the rows across all three replicas again:

Replicas 0 and 1 now match. Replica 2 still does not, because nothing touched it. That is exactly the limitation listed in the comparison table earlier: an external script synchronizes only at the moment it runs, and only what you tell it to.

Step 15 — Clean Up

Because everything lives in one namespace, deleting it removes the Pods, the Service, the Secret, the ConfigMap, the StatefulSet, and the PVCs in a single command.

Confirm that the PersistentVolumes were released as well:

With a Delete reclaim policy the volumes disappear on their own. If your StorageClass uses Retain, they will remain in a Released state and you must remove them manually. This is the leftover-storage problem from the previous article, one level deeper.

Summary

In this article you learned where the boundary of a StatefulSet's responsibility lies:

1. A StatefulSet manages Pod identity and storage binding. It never copies data between replicas.
2. Three duties stay with you: data cloning and synchronization, storage that is reachable from every node, and external backups for disaster recovery.
3. Data initialization can be done with init containers, the database image's own initialization directory, or both.
4. Ongoing consistency needs application-level replication. External scripts only close the gap at the moment they run.
5. The five key spec fields are replicas, serviceName, selector, template, and volumeClaimTemplates.
6. PVC names follow <volumeClaimTemplateName>-<statefulSetName>-<ordinal> and are permanently tied to one Pod.
7. Connecting to the headless Service reaches an arbitrary replica. For a database, name a specific Pod instead.
8. Divergence between replicas is silent. Every Pod stays Running and Ready while the data drifts apart.

In the next article we will look at releasing a new version of an application deployed as a StatefulSet, and at the update strategies that control how those ordered rollouts behave.