Updating StatefulSets — Rolling Updates, Canaries, and Rollbacks

In the previous article you learned that a StatefulSet scales in a strict order. That same ordering rules how a new version of your application is released. Changing the Pod template (.spec.template) triggers a rollout, exactly as it does for a Deployment, but the way that rollout proceeds is quite different.

This article covers the two update strategies, the partition field that turns a rollout into a canary release, what a broken rollout looks like and how to escape it, and how to roll back safely when the thing you are upgrading holds data.

We will cover the following topics:

1. The RollingUpdate and OnDelete strategies
2. Why rollouts run in reverse ordinal order
3. Staged and canary releases with .spec.updateStrategy.rollingUpdate.partition
4. How a failed rollout stalls, and the only way to recover it
5. Rolling back, and why a database makes rollback dangerous

Technical Requirements

1. A running Kubernetes cluster with a default StorageClass. Docker Desktop, minikube, or a cloud cluster all work.
2. kubectl installed and configured.
3. Some patience. Because each Pod is replaced one at a time and each waits to become ready, a rollout of three replicas takes a couple of minutes.

Core Concepts

The Two Update Strategies

You choose a strategy with the .spec.updateStrategy.type field. There are exactly two options.

RollingUpdate is the default. It terminates and recreates Pods one at a time, in order, and waits for each replacement to be Running and Ready before moving on. It resembles the Deployment strategy of the same name, but it is not identical: a Deployment can bring up several new Pods at once and can run old and new Pods side by side under maxSurge. A StatefulSet does neither. There is exactly one Pod in flight at any moment.

OnDelete is the legacy behaviour from before Kubernetes 1.7, and it is still genuinely useful. Under this strategy the controller updates the template but does not recreate any Pod. Nothing happens until you delete a Pod by hand, and only then does the replacement pick up the new template.

OnDelete exists for workloads where a human or an external process must check something between replicas. If you run a Cassandra or etcd cluster, you may want to confirm that the replacement node rejoined the cluster correctly before touching the next one. You can express similar checks with postStart and preStop lifecycle hooks under RollingUpdate, but the hooks need careful error handling and are harder to get right.

Rollouts Run in Reverse Order

A RollingUpdate respects every guarantee you learned about scaling. Pods are replaced from the highest ordinal downwards: chat-history-2 first, then chat-history-1, and chat-history-0 last.

This order is deliberate. Ordinal 0 is conventionally the most important replica, often the primary in a replicated database. Updating it last means the riskiest change happens after the lower-stakes replicas have already proved the new version works.

What Happens When a Rollout Fails

If a rollout goes wrong, the controller tries to keep the application consistent. Pods already updated stay on the new version; Pods not yet reached stay on the old one. The StatefulSet stops and waits.

Here is the part that surprises people. If a replacement Pod never becomes Running and Ready, the rollout halts and waits for manual intervention. Re-applying the previous, working manifest is not enough. The controller will not proceed, because it is still waiting for that failed Pod to become ready, and a Pod that cannot start will never satisfy it.

The only way out is to delete the failed Pod yourself. Once it is gone, the controller recreates it from whatever template is current, and the rollout unblocks. We will reproduce this exact situation later in the article, because reading about it is far less convincing than watching it happen.

Staged Rollouts with partition

The .spec.updateStrategy.rollingUpdate.partition field is what makes canary releases possible. The rule is simple:

Every Pod whose ordinal is lower than partition is left alone. Those Pods keep the old template, and even if you delete one, it comes back on the old version.

The default is 0, which is why an ordinary rollout touches every Pod: every ordinal is greater than or equal to zero. With three replicas, other values behave like this:

The workflow is to set partition equal to replicas, apply your new template so it is staged but inert, then walk the number down, verifying at each step.

Rolling Back a Stateful Application

Technically, rollback works the same as for a Deployment: kubectl rollout undo reverts to the previous revision. For StatefulSets, however, the declarative approach is strongly preferred. Keep your manifests in a source repository, revert the commit, and apply again. This fits the Infrastructure as Code model and usually runs through a CI/CD pipeline rather than an operator typing commands.

There is also a practical reason. Running kubectl rollout undo on an object you manage with kubectl apply leaves the stored last-applied-configuration annotation out of step with the live object, and your next apply can behave unexpectedly. Kubernetes warns you about this when you do it.

The deeper danger is data. Downgrading a container image does not downgrade your data. If the new version migrated a schema, changed an on-disk format, or wrote records the old version cannot read, rolling the image back leaves the old binary staring at data it does not understand. The application will refuse to start, or worse, start and misbehave.

Before you roll back anything stateful, ask whether a downward migration exists. If it does not, the safe path is to roll forward with a fix, not backward to the previous image.

Hands-On: Kubernetes Commands

Watch a rollout in progress

Prints a line per phase. It takes noticeably longer than a Deployment because Pods go one at a time.

List the stored revisions

Each revision is backed by a ControllerRevision object holding a snapshot of the Pod template.

Check which image a Pod is actually running

Reading the Pod, not the StatefulSet, is what tells you whether the update reached that replica.

See the recreation order in the events

The controller logs each delete and recreate, so you can confirm the reverse ordering afterwards.

Change the partition imperatively

Useful for a quick experiment, though editing the manifest is clearer and easier to review.

Change the image imperatively

Acceptable for testing. For production, change the manifest and apply it.

Roll back to the previous revision

Reverts the Pod template. It does not, and cannot, revert your data.

Step-by-Step Example

The Scenario

We will run a chat history database as a three-replica PostgreSQL StatefulSet named chat-history, starting on postgres:17.0. A chat-shell Pod gives us a client. We will write a message, then carry it through a full rolling update, a staged canary release, a deliberately broken rollout, and a rollback, checking after each one that the message is still there.

We use PostgreSQL patch versions (17.0, 17.2, 17.4) on purpose. They share an on-disk format, so the same data directory works with all of them. Later we will use a major version change to break things intentionally.

Step 1 — Create the Namespace, Secret, and Service

The credentials decode to chatadmin and Ch@tH1st2026.

The headless Service gives each replica its stable DNS name, as covered in the previous article.

Step 2 — Create the StatefulSet

Note that updateStrategy and partition are written out explicitly even though these are the default values. The book-standard advice applies here: spelling out the defaults makes the manifest self-documenting, and it gives you an obvious place to edit when you stage a rollout.

Now the client Pod. It runs a newer PostgreSQL image only because we need the psql binary; it is not part of the StatefulSet.

Confirm the starting version:

Step 3 — Write Data We Can Track

Create a database and a row on chat-history-0. Everything that follows will check that this data survives.

Remember from the earlier article that this data exists only on chat-history-0. The other two replicas have their own empty volumes.

Step 4 — A Full Rolling Update

Edit the image in the manifest:

The status output shows the rollout advancing one Pod at a time:

If you watch the Pods in another terminal with kubectl get pods -n chatops -w, the reverse order is unmistakable. Sampled every few seconds, it looks like this:

Ordinal 2 goes first, ordinal 0 last, and only one Pod is ever unavailable. Verify the result and that the data came through:

The row written under 17.0 is readable under 17.2. The Pod was destroyed and rebuilt, but it re-attached the same PersistentVolumeClaim.

Step 5 — Stage a Rollout Without Changing Anything

Now a canary release to postgres:17.4. First set partition equal to the replica count so that applying the new template changes nothing:

Change the image to postgres:17.4 at the same time, then apply:

Check the Pods. All three still run the old image, and none restarted:

The new template is stored and ready, waiting for you to lower the number. This is what staging a rollout means.

Step 6 — Release One Canary

Lower partition to 2. Only ordinal 2 qualifies for an update:

Notice the wording: 1 new pod, not 3. Now compare the images:

Run a smoke test against the canary and against an untouched replica:

Two versions running side by side in one StatefulSet, each addressable by its own stable DNS name. This is where you would run your real verification: application tests, query checks, log inspection.

Step 7 — Complete the Phased Rollout

The canary looks healthy, so bring partition back to 0:

The status starts at "1 out of 3" because the canary was already updated. Only the two remaining Pods are replaced. Between the canary and this step you could have paused at partition: 1 for a smaller phase.

All three replicas are now on 17.4, and the original message is still there:

Step 8 — Break the Rollout on Purpose

Now let's see what a stuck rollout really looks like. Change the image to postgres:16, a major version downgrade. PostgreSQL 16 cannot read a data directory written by version 17, so the container will fail to start.

Sampled every five seconds, the rollout goes nowhere:

Ordinals 0 and 1 are untouched and still serving on 17.4. Only the first Pod in the rollout is broken, and the controller will not move past it. The logs say exactly why:

This is the real-world shape of the rollback danger described earlier. The image went backwards; the data on disk did not.

Step 9 — Discover That Re-Applying Does Not Help

The instinctive fix is to put the working image back and apply again. Try it:

A full minute later, nothing has improved:

The Pod is still on postgres:16. The StatefulSet accepted the new template but will not act on it, because it is waiting for chat-history-2 to become ready first. The Pod cannot become ready. The rollout is deadlocked.

Step 10 — Recover With Manual Intervention

Delete the failed Pod yourself. That is the only way out:

The replacement is created from the current template and starts normally:

The data on chat-history-0 was never at risk, because the failure never reached that replica. Confirm it:

This is a good argument for canary releases on stateful workloads. Had we rolled out the bad image with partition at 3 and stepped down carefully, the damage would have been identical: contained to one replica, with the primary untouched.

Step 11 — The OnDelete Strategy

Switch strategies to see the manual mode. Contrary to some older documentation, the updateStrategy field can be changed on a live StatefulSet in current Kubernetes versions; you do not have to delete and recreate the object.

Now change the image and watch nothing happen:

The StatefulSet's template has changed, but no Pod has:

Delete a single Pod and only that one picks up the new template:

You are now in complete control of the pace, and you can run any verification you like between replicas. One consequence to know about: kubectl rollout status does not work under this strategy.

Switch back to RollingUpdate, and the controller immediately brings the remaining Pods to the template version:

Step 12 — Revisions and Rollback

Every template change stores a revision:

Roll back to the previous revision:

Kubernetes hands you an important warning:

Read that last sentence again. Kubernetes itself is telling you to prefer the declarative route. The rollback works, and the Pods return to the previous image in the usual reverse order:

And the message written back on 17.0 has survived every one of these operations:

In a real project you would not run rollout undo at all. You would revert the commit that changed the image, let the pipeline apply the manifest, and keep the cluster and the repository in agreement.

Step 13 — Clean Up

One command removes the Pods, the Service, the Secret, the StatefulSet, and the claims.

Check that no volumes were left behind.

Summary

In this article you learned how to release a new version of a stateful application:

1. Changing .spec.template starts a rollout, exactly as with a Deployment.
2. RollingUpdate replaces Pods in reverse ordinal order, one at a time, waiting for each to be Ready. Ordinal 0 is updated last.
3. OnDelete updates the template but recreates nothing until you delete a Pod yourself. Use it when a manual check belongs between replicas.
4. partition protects every ordinal below it. Set it equal to replicas to stage a rollout, then step it down for a canary and phased release.
5. A failed rollout stalls. Re-applying the working manifest does not fix it. You must delete the failed Pod manually.
6. kubectl rollout status only works under RollingUpdate.
7. updateStrategy can be changed on a live StatefulSet in current Kubernetes versions.
8. Prefer declarative rollbacks: revert the commit and apply. Kubernetes itself warns against rollout undo on objects managed with apply.
9. Downgrading an image does not downgrade your data. Without a downward migration, roll forward with a fix instead of backward to the old version.

In the next article we will gather the practical guidance from this whole chapter into a set of StatefulSet best practices.