Kubernetes StatefulSets: Managing Stateful Applications

Overview

Most applications deployed in Kubernetes are stateless: each replica is identical, interchangeable, and holds no local data. If you restart one, another takes its place without any difference to the user. A Deployment is the perfect controller for this kind of workload.

However, some applications are fundamentally stateful. A database cannot simply be replaced by a fresh copy — it owns data that must survive restarts. A distributed cache node has a specific slot range. A message broker replica has its own queue partition. These applications require stable identity, stable storage, and predictable ordering.

Kubernetes addresses this with StatefulSets. A StatefulSet is a workload controller that maintains a sticky identity for each of its Pods. No matter how many times a Pod is rescheduled, it keeps its name, its DNS entry, and its storage volume.

Core Concepts

Stateless vs Stateful — The Fundamental Difference

To understand why StatefulSets exist, it helps to contrast them directly with Deployments:

When Should You Use a StatefulSet?

Use a StatefulSet when your application requires at least one of the following:

1. Persistent, Pod-specific storage — each instance must own its own data that must not be mixed with another instance's data (e.g., a database replica).
2. Stable, predictable Pod names — other services or configuration must address a specific Pod by a fixed DNS name (e.g., postgres-0.postgres-headless).
3. Ordered, graceful deployment and scaling — each Pod must be fully running before the next one starts (e.g., a Raft consensus cluster needs a quorum before adding new members).
4. Stable network identity across restarts — a Pod that crashes and is rescheduled still gets the same hostname and DNS name.

Common production use cases:

1. Relational databases: PostgreSQL, MySQL
2. NoSQL databases: MongoDB, Cassandra, CockroachDB
3. Message brokers: Apache Kafka, RabbitMQ
4. Distributed caches: Redis Cluster, Memcached
5. Search engines: Elasticsearch, OpenSearch
6. Coordination services: Apache ZooKeeper, etcd

The Headless Service

A StatefulSet requires a Headless Service. A regular Service assigns a virtual ClusterIP and load-balances requests across all Pods. A Headless Service — created by setting clusterIP: None — skips this virtual IP entirely and instead creates an individual DNS A record for each Pod.

This is how each Pod in a StatefulSet gets its own stable DNS address, in the form:

For example, the primary node of a PostgreSQL StatefulSet named postgres with headless service postgres-headless in the default namespace is reachable at:

This predictable address is what other services — such as your ASP.NET Core API — use to connect directly to the primary database node rather than being randomly load-balanced to any replica.

volumeClaimTemplates — Per-Pod Persistent Storage

In a Deployment, you attach a single PersistentVolumeClaim that all Pods share. In a StatefulSet, you define a volumeClaimTemplates section instead. Kubernetes automatically creates a separate PersistentVolumeClaim for each Pod when it is first scheduled.

For a StatefulSet with 3 replicas and a volume named postgres-data, Kubernetes creates:

1. postgres-data-postgres-0 — owned exclusively by postgres-0
2. postgres-data-postgres-1 — owned exclusively by postgres-1
3. postgres-data-postgres-2 — owned exclusively by postgres-2

Critically, if a Pod is deleted and rescheduled, it re-attaches to the same PVC — the data is never lost even if the Pod moves to a different node.

Ordered Pod Management

By default, StatefulSets use OrderedReady Pod management. This means:

1. Scale up: Pods are created one at a time in ascending order — pod-0 must be Ready before pod-1 starts.
2. Scale down: Pods are terminated in descending order — pod-2 is deleted before pod-1.
3. Updates: The StatefulSet rolls out updates from the highest ordinal backward.

This ordering matters for clustered software. A Kafka broker, for example, needs the first node (the controller) healthy before followers join.

For applications that do not require ordered startup, you can set podManagementPolicy: Parallel to speed up scaling operations.

Update Strategies

StatefulSets support two update strategies:

1. RollingUpdate (default) — Pods are updated one at a time, highest ordinal first.
2. OnDelete — Pods are only updated when you manually delete them. Useful when you need full control over the upgrade sequence.

Hands-On: Kubernetes Commands

List StatefulSet Pods (ordered)

StatefulSet Pods always appear with their ordinal suffix. The -l flag filters by label.

Expected output:

Describe a StatefulSet

Inspect the StatefulSet details including volume claim templates and update strategy:

Verify per-Pod PersistentVolumeClaims

Each Pod should have its own PVC. You can confirm this with:

Expected output:

Resolve a Pod's DNS name from inside the cluster

Test that the headless service DNS records resolve correctly by running a temporary Pod:

Connect to a specific StatefulSet Pod

Because each Pod has a predictable name, you can open a shell directly into the primary node:

Scale a StatefulSet

Scaling up adds new Pods in order. Scaling down removes from the highest ordinal first:

Watch the ordered startup

Use -w to watch the ordered creation of Pods in real time:

Step-by-Step Example

The Scenario

We will deploy a 3-replica PostgreSQL 16 cluster using a StatefulSet. An ASP.NET Core 10 Order API Deployment will then connect to the primary database node (postgres-0) using the stable DNS name provided by the Headless Service.

Step 1 — Create the Secret for Database Credentials

Never embed credentials directly in a StatefulSet manifest. Store them in a Secret and reference them via environment variables.

Step 2 — Create the Headless Service

The Headless Service must be created before the StatefulSet. It is what enables the per-Pod DNS records. Setting clusterIP: None is what makes it headless.

Step 3 — Create the StatefulSet

The StatefulSet references the headless service via serviceName. The volumeClaimTemplates section defines the PVC template — Kubernetes will automatically create one PVC per Pod named postgres-data-postgres-0, postgres-data-postgres-1, and postgres-data-postgres-2.

Step 4 — Watch the Ordered Pod Creation

Run this command and observe how Kubernetes creates postgres-0 first and waits for it to become Ready before creating postgres-1, and so on:

You will see output like this over approximately 1–2 minutes:

Step 5 — Deploy the ASP.NET Core Order API

The Order API Deployment connects directly to postgres-0 — the primary node — using its stable DNS name. Because the DNS name is predictable and permanent, the connection string in the environment variable never needs to change, even if the postgres-0 Pod is rescheduled to a different node.

Step 6 — Verify the DNS-Based Connection

From inside any Pod in the cluster, verify that the headless service resolves to individual Pod IPs — one DNS record per Pod, not a single load-balanced IP:

You will see a separate Address line for each of the three PostgreSQL Pods — not a single virtual IP. This confirms the headless service is working correctly.

Step 7 — Simulate a Pod Failure and Verify Data Persistence

Delete postgres-0 and watch Kubernetes recreate it with the same name and the same PVC:

Notice that the new postgres-0 Pod reattaches to the existing postgres-data-postgres-0 PVC. All data written before the deletion is still there. This is the fundamental guarantee of a StatefulSet.

Step 8 — Verify PVC Retention After Pod Deletion

PVCs created by a StatefulSet are not deleted when the Pod is deleted, and by default they are not deleted even when the StatefulSet itself is deleted. This protects your data from accidental loss.

To permanently delete the data, you must delete the PVCs manually:

Summary

StatefulSets are the right tool whenever your application needs identity, storage, or ordering guarantees that a Deployment cannot provide. Here is a concise summary of everything covered:

1. A StatefulSet gives each Pod a stable ordinal name (pod-0, pod-1...), a dedicated PersistentVolumeClaim, and a predictable DNS record — all of which survive Pod restarts and rescheduling.
2. A Headless Service (clusterIP: None) is required. It creates individual DNS A records for each Pod, enabling direct addressing via pod-name.service-name.namespace.svc.cluster.local.
3. volumeClaimTemplates automatically provision one PersistentVolumeClaim per Pod. PVCs are not deleted when Pods or even the StatefulSet is deleted — data is preserved.
4. Pods start and stop in strict order by default — essential for clustered systems that require a quorum or a designated primary node.
5. Use a StatefulSet for databases, message brokers, distributed caches, search engines, and coordination services — anything that holds state unique to each replica.
6. Use a Deployment for your application tier (APIs, frontends, workers) — these are stateless and benefit from the simpler Deployment controller.

The pattern of pairing a stateless ASP.NET Core Deployment with a stateful PostgreSQL StatefulSet is one of the most common and robust architectures in production Kubernetes environments.