Kubernetes Canary Deployments

A Canary deployment is a release strategy where you roll out a new version to a small subset of users first. If the new version is healthy, you gradually increase its share of traffic until it serves all requests. If something goes wrong, you scale the canary back to zero — only a small fraction of users were ever affected.

The name comes from the "canary in a coal mine" — a small, early warning signal. In Kubernetes, the canary is a small Deployment running the new version alongside the larger stable Deployment. Because both Deployments share the same app label, a single Service load-balances traffic across all Pods — the ratio of stable-to-canary Pods determines the traffic split.

Compared to Blue/Green (instant switch) and Rolling Updates (automatic gradual replacement), Canary gives you the most control over how much traffic the new version receives and how fast you ramp it up. This makes it ideal for high-traffic services where even a brief spike in errors is unacceptable.

Core Concepts

Step 1: How Canary Works — The Big Picture

The strategy uses two Deployments and one Service:

1. Stable Deployment — Runs the current production version (e.g., v1.0) with a higher replica count (e.g., 4 replicas).
2. Canary Deployment — Runs the new version (e.g., v2.0) with a lower replica count (e.g., 1 replica).
3. Service — Selects Pods based on the shared app label only. Because both Deployments have app: booking-api, the Service sends traffic to all 5 Pods (4 stable + 1 canary).

With this setup, roughly 20% of traffic goes to the canary (1 out of 5 Pods). You control the traffic ratio by adjusting replica counts.

Step 2: Traffic Splitting by Replica Ratio

Kubernetes Services distribute traffic roughly evenly across all matching Pods. This means the traffic percentage is approximately:

Common progression patterns:

Note: this is a rough approximation. Kubernetes round-robin distribution is not perfectly even, but it is close enough for most canary use cases.

Step 3: The Label Strategy

The critical difference between Canary and Blue/Green is how the Service selects Pods:

1. Blue/Green: The Service selects on app + version. Only one environment receives traffic at a time.
2. Canary: The Service selects on app only. Both Deployments receive traffic simultaneously. An additional track label (track: stable or track: canary) helps you manage the Deployments with kubectl, but the Service does not use it.

Step 4: Monitoring the Canary

The point of a canary is to observe the new version under real traffic before giving it more. Key things to monitor:

1. Error rate: Are 5xx responses increasing?
2. Latency: Is the canary slower than the stable version?
3. Pod restarts: Is the canary crashing?
4. Resource usage: Is CPU or memory unexpectedly high?

Compare these metrics between stable and canary Pods. If the canary looks healthy after an observation window (e.g., 10–30 minutes), increase its replica count.

Step 5: When to Use Canary

Step 6: Canary vs Blue/Green vs Rolling Update

Hands-On: Kubernetes Commands

Deploy the Stable Version

Create the Service

Deploy the Canary

Check All Pods Across Both Deployments

Check the Service Endpoints (Should Include Both Stable and Canary)

Scale Canary Up (Increase Traffic)

Scale Stable Down (Shift Traffic Toward Canary)

Promote Canary to Full Production

Rollback — Remove the Canary

Clean Up After Promotion

Step-by-Step Example

Scenario 1: Deploy a Canary and Observe

We have a .NET 10 Booking API running in production. A new version (2.0) includes a rewritten availability-check algorithm. We want to expose it to a small fraction of traffic first to make sure latency stays low.

1. Deploy the Stable Version (v1.0) with 4 Replicas

2. Create the Service

Notice the selector uses only app: booking-api — it intentionally does not include the track label. This is what allows both Stable and Canary Pods to receive traffic.

3. Verify Stable Is Running

You should see 4 Pods, all with track=stable.

4. Deploy the Canary (v2.0) with 1 Replica

5. Confirm Traffic Is Split

You should see 5 Pods total: 4 with track=stable and 1 with track=canary.

The endpoints list should contain 5 IP addresses. Approximately 20% of traffic now goes to the canary Pod.

Scenario 2: Gradually Promoting the Canary

After monitoring the canary for 15 minutes, error rates and latency look normal. Time to increase canary traffic.

6. Scale Canary to 2 Replicas (~40% Traffic)

Now 2 out of 5 Pods run the canary. Monitor for another observation window.

7. Scale Canary to 3 Replicas (~60% Traffic)

8. Full Promotion — Canary Becomes Production

All traffic now goes to version 2.0. Once you are confident, clean up the old Deployment:

Scenario 3: Canary Rollback

If at any point the canary shows problems — elevated errors, increased latency, or Pod crashes — immediately scale it to zero:

9. Emergency Rollback

All traffic returns to the stable Pods. If you had reduced stable replicas, scale them back up:

The canary Deployment still exists with 0 replicas. You can debug, fix, rebuild the image, and try again without deleting anything.

Summary

1. Canary deployment sends a small percentage of traffic to the new version by running it with fewer replicas alongside the stable version.
2. The Service selector matches only the app label, so it load-balances across both stable and canary Pods.
3. Traffic ratio is controlled by the replica counts of each Deployment. More canary replicas = more canary traffic.
4. Promotion is gradual — scale canary up and stable down in steps, with an observation window at each step.
5. Rollback is fast — scale the canary to 0 replicas and restore the stable replica count.
6. The track label (track: stable / track: canary) helps you manage and query Pods, but is intentionally excluded from the Service selector.
7. Best for: high-traffic services where you want to limit the blast radius of a new release and observe real production behaviour before full promotion.