Introducing StatefulSets Deploying Stateful Applications in Kubernetes

In the previous chapter, we used Deployment objects to run stateless workloads. Stateless applications are easy to run in the cloud: every Pod replica handles a request in exactly the same way, and no replica depends on what happened in a previous request. All you really need to worry about is load balancing.

The hard part of running applications in a cluster is managing state. By state, we mean any stored data that an application needs to serve requests and that those requests can modify. The most common example is a database, such as a relational PostgreSQL database or a NoSQL MongoDB database.

Kubernetes has a dedicated object for running stateful workloads: the StatefulSet. When you work with StatefulSets, you will almost always also work with PersistentVolumes, which were covered in the storage chapter. In this article, you will learn what a StatefulSet is, how it differs from a Deployment, what its limitations are, and how to deploy a PostgreSQL StatefulSet together with an ASP.NET Core API that talks to it.

In this article, we will cover the following topics:

1. What "state" actually means for an application
2. How state is managed in plain containers
3. How state is managed in Kubernetes Pods with PVs, PVCs, and StorageClasses
4. Why a Deployment is the wrong tool for a database
5. The StatefulSet object and how it differs from a Deployment
6. The limitations of StatefulSets

Technical Requirements

For the hands-on part of this article, you will need the following:

1. A running Kubernetes cluster. A local cluster such as Docker Desktop or minikube is enough, but a multi-node cloud cluster shows the concepts best. The cluster must be able to create PersistentVolumeClaims dynamically. The default StorageClass that ships with Docker Desktop and minikube is sufficient.
2. kubectl installed and configured to talk to your cluster.
3. Docker and the .NET 10 SDK if you want to build the sample API image yourself.

Core Concepts

What Is "State", Really?

Before we can talk about stateful workloads, we need a clear definition. State is data that is persisted and that user requests can change. If a previous request can influence the result of the current request, your component holds state.

Consider a web server that only serves a static HTML page. There is persisted data on disk (the HTML files), but no request can modify it. Every request gets the same answer, so this is not state. The same goes for the server's configuration files and, from the end user's point of view, its log files.

Now consider a web server that keeps user sessions and remembers whether a user is logged in. Depending on that information, the server returns different pages. This is state. But where the state lives matters:

1. If the web server stores sessions in a file inside its own container, the web server itself becomes a stateful component. This is usually poor design.
2. If the web server stores sessions in a database or a Redis cache running in a separate container, the web server stays stateless, and the database or Redis container becomes the stateful component.

Remember: almost every application as a whole is stateful. That does not mean every component has to be. Good cloud-native design keeps most components stateless and isolates the state in a few well-managed places.

Why Stateful Is Harder Than Stateless

In a classic three-tier application, all state lives in the database tier. Nothing about that is special. For high availability, you would add a failover replica. For performance, you would scale vertically by buying a bigger server. Eventually you might introduce a clustered database with data sharding (horizontal partitions of the data). But from the web server's perspective, the database is still just one connection string.

In a distributed, container-based cluster, the database itself becomes a set of containers that can be restarted, rescheduled, and moved between nodes. Each replica now needs its own identity and its own storage, and the replicas need to find each other by name. That is the complexity StatefulSets are built to manage.

Managing State in Plain Containers

Imagine you run a single PostgreSQL server in a container. The first thing you notice is that every restart gives you a fresh, empty database. Containers are ephemeral: their filesystem is thrown away when the container is removed.

The fix is a volume. A volume is a directory on the host, an external disk, or an NFS share that is mounted to a path inside the container's filesystem. Whatever you write to that path survives container restarts. If you configure PostgreSQL to store its data files on the mounted path, the container stays ephemeral but the data does not.

You can see this with plain Docker before we move to Kubernetes:

The -v postgres-demo-data:/var/lib/postgresql/data flag mounts a named volume at PostgreSQL's data directory. If you remove the container and start a new one with the same volume, your databases are still there.

Managing State in Kubernetes Pods

Kubernetes extends the container volume idea with three dedicated storage objects. You have met these in the storage chapter, so here is a short refresher:

With a StorageClass in place, storage provisioning becomes fully dynamic: you create a PVC, the provisioner creates a PV (for example, a cloud-managed disk), and the Pod mounts it. Containers inside the same Pod can also share one PV.

You can check which StorageClasses your cluster offers with this command:

On Docker Desktop you will see something like this:

The class marked (default) is used whenever a claim does not name a StorageClass explicitly. A default StorageClass is what makes the example later in this article work without any extra setup.

Why a Deployment Is the Wrong Tool for a Database

Storage for a single Pod is only half of the problem. What happens when you run multiple replicas of a database? Let's think through using a Deployment to run three PostgreSQL Pods:

1. You add a PVC to the Pod template so the data is persisted. So far, so good.
2. The Deployment creates three Pods. But a Deployment has one Pod template with one PVC reference, so all three Pods try to mount the same volume. Most disks cannot be mounted read-write by several nodes at once, and even if they could, three PostgreSQL processes writing to one data directory would corrupt it.
3. Suppose you work around that and give each Pod its own disk. Now you have three completely separate PostgreSQL servers with three different sets of data. That is not high availability.
4. If you expose them behind one Service, every request may hit a different Pod and see different data.

The real solution is a replicated setup (a primary with replicas) or a sharded cluster. Both need every PostgreSQL Pod to have a unique, stable identity and a predictable network name so the members can find each other and so clients know which one is the primary. Deployments give Pods random names and treat them as interchangeable, so they cannot provide this.

Always weigh the pros and cons before running stateful components inside Kubernetes. A managed cloud database is often the simpler choice. When you do decide to run state in the cluster, the StatefulSet is the object to use.

The StatefulSet Object

A StatefulSet is very similar to a Deployment. It manages a set of Pods based on a Pod template, it can be scaled up and down, and it can roll out new versions. The difference is that a StatefulSet gives each Pod a persistent, unique identity and guarantees ordering. Pod replicas are not interchangeable.

The identity of each Pod is kept across restarts, rescheduling, and rollouts, and it consists of:

1. A sticky Pod name in the form <statefulSetName>-<ordinal>, for example ledger-postgres-0, ledger-postgres-1, ledger-postgres-2.
2. A stable cluster DNS name for each Pod, provided through a headless Service.
3. A dedicated PersistentVolumeClaim per Pod, created from a volumeClaimTemplates section in the StatefulSet spec and always re-attached to the Pod with the same name.

Because the PVCs are created from a template, the whole storage workflow becomes automatic: you create the StatefulSet, and the StatefulSet controller creates one PVC per Pod, the StorageClass provisions a PV for each claim, and each Pod mounts its own disk.

One important detail: the DNS names of the Pods stay the same, but their cluster IP addresses are not guaranteed to. Always connect to individual StatefulSet Pods by DNS name, never by IP.

Use a StatefulSet for applications that require one or more of the following:

1. Persistent storage managed by the cluster (the main use case, but not the only one)
2. Stable, unique network identifiers (DNS names) for each replica
3. Ordered, graceful deployment and scaling
4. Ordered, automated rolling updates

StatefulSet vs Deployment

The table below summarises the key differences:

The Headless Service

A StatefulSet requires a headless Service: a Service with clusterIP: None. A regular ClusterIP Service returns one virtual IP and load-balances behind it. A headless Service instead returns the IP addresses of all matching Pods as individual DNS A records, and it also creates a DNS record for each StatefulSet Pod in the form:

For our example this will be ledger-postgres-0.ledger-postgres-headless.default.svc.cluster.local, or simply ledger-postgres-0.ledger-postgres-headless from within the same namespace. The StatefulSet links to the Service through the .spec.serviceName field. Headless Services were covered in detail in the Services chapter.

Limitations of StatefulSets

StatefulSets solve real problems, but they come with a few things you must keep in mind:

1. Storage is not created out of thin air. The StatefulSet creates PVCs, but something must fulfil them: either a dynamic provisioner via a StorageClass, or PVs you create manually beforehand.
2. Storage is left behind. Scaling down or deleting a StatefulSet does not delete its PVCs. This is deliberate, because your data is valuable. But you must clean up unused claims yourself, or they will accumulate and cost money.
3. A headless Service is required to give Pods their stable network names. You have to create it yourself.
4. Deletion is not ordered. When you delete a StatefulSet, Pods are not guaranteed to terminate in reverse order. For a clean shutdown, scale to zero first, then delete.
5. Default updates can get stuck. The default rolling update strategy can leave the StatefulSet in a broken state that needs manual repair, for example if a new Pod never becomes ready. Update strategies are covered in the next article.

Hands-On: Kubernetes Commands

These are the commands you will use most often when working with StatefulSets.

List StatefulSets

Shows every StatefulSet in the current namespace with its ready replica count. sts is the short name.

List StatefulSet Pods with their ordinals

Filter by the StatefulSet's label and add -o wide to see which node each Pod landed on.

Describe a StatefulSet

Shows the Pod template, the volume claim templates, the update strategy, and recent events.

List the PersistentVolumeClaims created by a StatefulSet

Each claim is named <volumeClaimTemplateName>-<podName>.

Scale a StatefulSet

Scaling up adds Pods in ascending order; scaling down removes them in descending order.

Watch Pods appear one by one

The -w flag streams changes, so you can see the ordered creation live.

Open a shell in a specific replica

Because names are deterministic, you can target an exact Pod without looking it up first.

Check rollout status

Works for StatefulSets just like for Deployments.

Delete a StatefulSet but keep its Pods

The --cascade=orphan flag removes only the controller object. This is useful during maintenance when you do not want any Pod to be touched.

Step-by-Step Example

The Scenario

We will build a small Ledger system with two components. The stateful component is a PostgreSQL StatefulSet named ledger-postgres with three replicas, each with its own disk. The stateless component is an ASP.NET Core API named ledger-api, deployed as a normal Deployment, that reads and writes ledger entries in the database using the stable DNS name of the first PostgreSQL Pod.

One honest note before we begin: the three PostgreSQL Pods in this example are three independent servers. Turning them into a real primary-replica cluster needs extra PostgreSQL configuration that is outside the scope of this introduction. The goal here is to see the StatefulSet guarantees in action: ordered creation, sticky names, per-Pod storage, and stable DNS.

Step 1 — Create the Secret for Database Credentials

We never hard-code passwords in a Pod template. The Secret below holds the application user's credentials, base64-encoded. The decoded values are ledgeradmin and L3dg3rP@ss2026. The official PostgreSQL image creates this user as the owner of the database on first start.

Step 2 — Create the Headless Service

The Service below has clusterIP: None, which makes it headless. It does not get a virtual IP. Instead, Kubernetes DNS will publish one A record per Pod that matches the app: ledger-postgres selector. Create this before the StatefulSet so the DNS records exist as soon as the Pods do.

Confirm that the Service has no cluster IP:

Step 3 — Create the StatefulSet

Now the main object. Read it top to bottom and notice the three parts that are new compared to a Deployment:

1. spec.serviceName points to the headless Service from Step 2.
2. The container mounts a volume named postgres-data at /var/lib/postgresql/data. The PGDATA variable points PostgreSQL to a subdirectory of that mount, which avoids problems with provisioners that place a lost+found folder at the root of a fresh disk.
3. spec.volumeClaimTemplates defines that volume as a 2 GiB ReadWriteOnce claim. One PVC will be created per Pod from this template.

The readiness and liveness probes use pg_isready, which returns success only once the server accepts connections. PostgreSQL initialises its data directory on first start, so the initial delays give it time to finish.

Step 4 — Watch the Ordered Creation

Immediately after applying, watch the Pods appear:

You will see ledger-postgres-0 go through Pending, ContainerCreating, and Running. Only after it reports 1/1 ready does ledger-postgres-1 appear, and only after that one is ready does ledger-postgres-2 follow. A Deployment would have started all three at once. Press Ctrl+C to stop watching once all three are ready. On a multi-node cluster, the -o wide flag also shows that the Pods are spread across nodes:

Now look at the storage the StatefulSet created for you:

You should see three claims, each bound to its own PersistentVolume:

Notice the naming pattern: <volumeClaimTemplateName>-<podName>. This is how the controller knows which claim belongs to which Pod when a Pod is recreated.

Step 5 — Verify the Stable DNS Names

The easiest place to test DNS is from inside another member of the StatefulSet, because the PostgreSQL image already contains the tools we need. Resolve the headless Service and then one individual Pod from ledger-postgres-1:

The first lookup returns three IP addresses, one per Pod, because the Service is headless. The second returns exactly one: the IP of ledger-postgres-0. That per-Pod name is what our API will use. Even if the Pod is rescheduled and gets a new IP, the name will keep resolving correctly.

You can go one step further and prove that one member can open a database connection to another by name:

If you prefer a separate debugging Pod, note that the nslookup in recent busybox images does not apply the cluster's DNS search list reliably. Use the fully qualified name instead:

Step 6 — Prove That Data Survives a Pod Restart

This is the moment where "containers are ephemeral, volumes are not" becomes visible. First, create a table and insert a row in ledger-postgres-0. At the same time, write a scratch file to /tmp, which is not on the volume.

Inside the Pod, run the following. The psql client picks up the user from the environment, so no password prompt appears when connecting locally:

Now delete the Pod. The StatefulSet controller will recreate it with the same name:

Once the new Pod is ready, check both pieces of data:

The first command fails: the container's filesystem was thrown away. The second command prints the Opening balance row: the data lived on the PVC named postgres-data-ledger-postgres-0, and the new Pod mounted exactly that claim because it has the same identity.

Step 7 — Build the ASP.NET Core Ledger API

Now let's add the stateless component. The API is a tiny .NET 10 minimal API that reads its connection details from environment variables and talks to PostgreSQL through the Npgsql package. Create the project and add the package:

Replace the contents of Program.cs with the following:

The API has no state of its own. Every request opens a connection, talks to PostgreSQL, and closes it. That is exactly why it can be a Deployment: any replica can serve any request.

Add a Dockerfile next to the project file. It uses a multi-stage build so the final image contains only the runtime:

Build the image:

On Docker Desktop, the Kubernetes cluster shares the local image store, so the image is immediately available. On minikube, load it with minikube image load ledger-api:1.0. On a cloud cluster, push the image to your container registry and update the image name in the Deployment below.

Step 8 — Deploy the Ledger API

The Deployment sets POSTGRES_HOST to ledger-postgres-0.ledger-postgres-headless: the stable per-Pod DNS name from Step 5. The credentials come from the same Secret the database uses, so there is a single source of truth for the password.

Compare the Pod names of the two workloads side by side:

The output makes the naming difference obvious:

Forward a local port to the Deployment and exercise the API. Open a second terminal for the curl commands:

The second call returns both the Opening balance row you inserted by hand in Step 6 and the new Coffee supplies entry. The stateless API and the manual SQL session both reached the same disk through the same stable name.

If you look at the API logs with kubectl logs deployment/ledger-api, you may see a line about libgssapi_krb5.so.2 not being found. Npgsql looks for that Kerberos library at startup and falls back to password authentication when it is missing. The message is harmless in this example.

Step 9 — Scale Down and Observe Leftover Storage

Scale the StatefulSet from three replicas to one and watch the order in which Pods disappear:

ledger-postgres-2 terminates first, then ledger-postgres-1. ledger-postgres-0, the Pod our API depends on, is untouched. Now check the claims:

All three PVCs are still there, still Bound, even though only one Pod exists. This is the "leftover storage" limitation in action. If you never scale back up, you are paying for two disks nobody uses. Scale back up to two and confirm that ledger-postgres-1 re-attaches to its old claim instead of getting a new one:

The PVC list is unchanged: same names, same volume IDs. The Pod found its old data directory waiting.

Step 10 — Clean Up

Follow the recommended order: scale to zero for a graceful shutdown, delete the objects, and then delete the PVCs explicitly, because Kubernetes will not do it for you.

Verify that nothing is left behind:

Summary

In this article you learned the foundations of running stateful workloads in Kubernetes:

1. State is persisted data that requests can modify. Static files and configuration are not state; user sessions and database rows are.
2. Containers are ephemeral. Volumes keep data alive across restarts. In Kubernetes, PVCs request storage, PVs provide it, and StorageClasses provision it dynamically.
3. A Deployment treats Pods as interchangeable and gives them random names, so it cannot run a replicated database properly.
4. A StatefulSet gives each Pod a sticky ordinal name, a stable DNS name through a headless Service, and its own PVC created from volumeClaimTemplates.
5. Pods are created in order (-0, -1, -2) and scaled down in reverse order.
6. Always address StatefulSet Pods by DNS name, never by IP.
7. StatefulSets do not delete PVCs when scaled down or deleted, do not guarantee ordered deletion, and require you to provide storage provisioning and a headless Service yourself.

In the next article, we will look at managing StatefulSets day to day and at releasing new versions of a stateful application safely with the available update strategies.