Why Services Exist — Pod Networking and the ClusterIP Service

Lesson 19 of 39 15 min read Updated

By now you can deploy applications on Kubernetes. You can build Pods, separate their configuration into ConfigMaps, and keep secrets out of your manifests. With those three pieces you can get an application running. But something important is still missing: you cannot yet reach it reliably.

This article explains why. It starts with an experiment that takes thirty seconds and changes how you think about Pods: delete one, recreate it from the exact same file, and watch its IP address change. Then we introduce the object that solves the problem, the Service, and cover the most widely used type, ClusterIP.

This is the first of three articles on Services. The second covers reaching your Pods from outside the cluster, and the third covers the probes that decide when a Pod is allowed to receive traffic.

We will cover the following topics:

  1. The four networking problems Kubernetes has to solve
  2. How Pods get IP addresses, and why those addresses are not yours to keep
  3. What a Service is, and how labels and selectors connect it to Pods
  4. How Service DNS names are built
  5. Creating and testing a ClusterIP Service
  6. How traffic is actually distributed across replicas

Technical Requirements

  1. A working Kubernetes cluster, local or cloud-based.
  2. kubectl configured to talk to it.
  3. Docker and the .NET 10 SDK if you want to build the sample image yourself.

Core Concepts

The Four Networking Problems

Networking is not an add-on in Kubernetes; it is one of its central responsibilities. There are four distinct problems, and it is worth knowing which one you are solving at any moment.

ProblemSolved by
Container to containerThe Pod itself. Containers in one Pod share a network namespace and reach each other over localhost.
Pod to PodThe cluster network. Every Pod gets a unique IP and can reach every other Pod directly, even across nodes.
Pod to ServiceThe Service object. It puts a stable name and address in front of a changing set of Pods.
Outside to ServiceNodePort, LoadBalancer, and Ingress. This is the subject of the next article.

This article is about the third one. It exists because of a property of the second one that surprises most beginners.

Who Assigns Which IP Address

A cluster uses three separate, non-overlapping address ranges, each managed by a different component.

  1. Pods are given addresses by a network plugin, almost always a Container Network Interface (CNI) plugin.
  2. Services are given addresses by the kube-apiserver.
  3. Nodes are given addresses by the kubelet or the cloud controller manager.

CNI is a specification from the Cloud Native Computing Foundation that standardises how a container's network interface is configured. Kubernetes does not implement the cluster network itself; it delegates to whichever plugin you install. Common open-source choices are Calico for fine-grained network policy, Flannel for a simple overlay, Cilium for eBPF-based security and observability, and Weave Net for easy encrypted setups. Managed clouds usually ship their own: AWS VPC CNI, Azure CNI, or VMware NSX-T.

Whichever plugin you use, the guarantee is the same: every Pod gets its own unique IP address. That address is private to the cluster. Typing it into a browser on your laptop reaches nothing.

Pod Addresses Are Not Yours to Keep

Here is the fact that makes Services necessary. A Pod's IP address is assigned when the Pod is created and is gone when the Pod is gone. Recreate the Pod from the identical manifest and you get a different address, because it is not the same Pod. It is a new one that happens to look like the old one.

Think about what that means for real applications:

  1. A microservice A that calls microservice B by IP breaks the moment B is rescheduled.
  2. An application holding a database connection string with a Pod IP loses its database on the next restart.
  3. A client caching the address of a Redis Pod talks to nothing after a node drain.

So the rule is simple and absolute: never hardcode a Pod IP address anywhere. Not in application code, not in a ConfigMap, not in a manifest. You need something that stays put while Pods come and go.

The Service Object

A Service is that something. It is a long-lived object that sits in front of a changing set of Pods and gives them one address and one name. Pods are meant to be disposable; Services are meant to stay. Even if you delete and recreate a Service, clients keep working, because they use its name, not its address.

Creating a Service attaches two things to it:

  1. A virtual IP address of its own, unique in the cluster.
  2. An internal DNS name, generated from the Service name, that never changes.

You can use either, but in practice you always use the name. It is predictable, and you can write it into configuration before the Service even exists.

How Service DNS Names Are Built

The fully qualified name of a Service follows one pattern:

<service-name>.<namespace>.svc.cluster.local

Only the first two parts vary. The svc.cluster.local suffix is fixed. A Service named voucher-api in the retail namespace is therefore always reachable at voucher-api.retail.svc.cluster.local from anywhere in the cluster.

Cluster DNS is configured with a search list, so shorter forms work too. From a Pod in the same namespace, plain voucher-api is enough. From another namespace you need at least voucher-api.retail. Use the short form in the same namespace and the fully qualified form when crossing namespaces, and you will avoid a whole class of confusing failures.

Labels and Selectors Are the Whole Mechanism

A Service does not reference Pods by name. It carries a selector, which is a set of labels, and it claims every Pod whose labels match. That is the entire link between the two objects, and it is why membership is dynamic.

The workflow is always the same:

  1. You create Pods and give them labels of your choosing.
  2. You create a Service whose selector matches those labels.
  3. The Service continuously watches for matching Pods and tracks their addresses.
  4. A client calls the Service by DNS name.
  5. The Service forwards the request to one of the matching Pods.

Two consequences follow. Adding a Pod with matching labels puts it into rotation automatically, with no change to the Service. And a selector that matches nothing produces a Service that exists, resolves in DNS, and refuses every connection.

How Traffic Is Distributed

People often describe Services as doing round-robin load balancing. That is a useful mental model but not literally true in most clusters, and the difference occasionally matters.

The component doing the work is kube-proxy, which runs on every node and programs kernel rules. It has two common modes:

  1. iptables, the default in most distributions, picks a backend using random selection with equal probability. Over many requests the split is even; over ten requests it will not be.
  2. IPVS is a real load balancer with real algorithms, including round-robin, least connections, and source hashing, plus session affinity and traffic shaping.

Check which mode your cluster uses before reasoning about traffic distribution:

kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep '^mode:'
mode: iptables

There is no separate proxy process per Service. A Service is a logical object; kube-proxy translates it into forwarding rules on each node.

Hands-On: Kubernetes Commands

See Pod IP addresses

The -o wide flag adds the IP and node columns, which the default output hides.

kubectl get pods -n retail -o wide

Show the labels on a Pod

These are what a Service selector has to match.

kubectl get pods -n retail --show-labels

List Services

svc is the accepted abbreviation.

kubectl get svc -n retail

Describe a Service

Shows the selector, the virtual IP, the ports, and the current backend addresses.

kubectl describe svc voucher-api -n retail

See which Pods a Service currently sends traffic to

EndpointSlices are the modern replacement for the older Endpoints object.

kubectl get endpointslice -n retail -l kubernetes.io/service-name=voucher-api

Reach a Service from your laptop, for testing only

This is a debugging tool. It stops the moment you press Ctrl+C.

kubectl port-forward svc/voucher-api -n retail 8080:80

Create a Pod and a Service in one command

Convenient for a quick check, but see the warning about it further down.

kubectl run scratch-nginx --image nginx --expose=true --port=80

Step-by-Step Example

The Scenario

We will run a voucher API with three replicas in a retail namespace. Each replica reports which Pod answered, so load balancing becomes visible in the output. A netshoot Pod gives us a shell with networking tools, because cluster-internal names cannot be resolved from your laptop.

Step 1 — Create the Namespace

apiVersion: v1
kind: Namespace
metadata:
name: retail
kubectl apply -f 1-retail-namespace.yaml

Step 2 — Build the Voucher API

A .NET 10 minimal API that reports its own identity. The values come from environment variables that Kubernetes fills in through the Downward API.

dotnet new web -n VoucherApi
cd VoucherApi

Replace Program.cs with the following:

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

string podName = Environment.GetEnvironmentVariable("POD_NAME") ?? Environment.MachineName;
string podIp = Environment.GetEnvironmentVariable("POD_IP") ?? "unknown";
string nodeName = Environment.GetEnvironmentVariable("NODE_NAME") ?? "unknown";
int served = 0;

app.MapGet("/healthz", () => Results.Ok("healthy"));

// Every reply names the Pod that produced it, which makes Service load balancing visible.
app.MapGet("/", () => Results.Ok(new
{
servedByPod = podName,
podIp,
node = nodeName,
requestsServedByThisPod = Interlocked.Increment(ref served)
}));

app.MapGet("/whoami", () => Results.Text($"{podName} ({podIp}) on {nodeName}\n"));

app.Run();

Add a Dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY . .
RUN dotnet publish VoucherApi.csproj -c Release -o /app

FROM mcr.microsoft.com/dotnet/aspnet:10.0
WORKDIR /app
COPY --from=build /app .
ENV ASPNETCORE_HTTP_PORTS=8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "VoucherApi.dll"]
docker build -t voucher-api:1.0 .

On Docker Desktop the cluster shares the local image store, so the image is available at once. On minikube run minikube image load voucher-api:1.0. On a cloud cluster, push to a registry and change the image name below.

Step 3 — Deploy Three Replicas

Note the labels block in the Pod template. Those labels, not the Deployment name, are what the Service will look for.

apiVersion: apps/v1
kind: Deployment
metadata:
name: voucher-api
namespace: retail
labels:
app: voucher-api
tier: backend
spec:
replicas: 3
selector:
matchLabels:
app: voucher-api
template:
metadata:
labels:
app: voucher-api
tier: backend
spec:
containers:
- name: voucher-api
image: voucher-api:1.0
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 8080
env:
# The Downward API lets each replica report its own identity, which is what
# makes Service load balancing visible in the responses.
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
requests:
cpu: "50m"
memory: "96Mi"
limits:
cpu: "300m"
memory: "256Mi"
readinessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
livenessProbe:
httpGet:
path: /healthz
port: http
initialDelaySeconds: 15
periodSeconds: 20
timeoutSeconds: 3
failureThreshold: 3
kubectl apply -f 2-voucher-api-deployment.yaml
kubectl rollout status deployment voucher-api -n retail

Step 4 — The Experiment: Watch an IP Address Disappear

Look at the current addresses:

kubectl get pods -n retail -o wide
NAME READY STATUS IP NODE
voucher-api-c7487d7f7-ggf67 1/1 Running 10.244.2.80 desktop-worker
voucher-api-c7487d7f7-n8v28 1/1 Running 10.244.2.79 desktop-worker
voucher-api-c7487d7f7-vszsl 1/1 Running 10.244.1.73 desktop-worker2

Now delete one and let the Deployment replace it:

kubectl delete pod voucher-api-c7487d7f7-ggf67 -n retail
kubectl rollout status deployment voucher-api -n retail
kubectl get pods -n retail -o custom-columns='NAME:.metadata.name,IP:.status.podIP'
voucher-api-c7487d7f7-n8v28 10.244.2.79
voucher-api-c7487d7f7-vszsl 10.244.1.73
voucher-api-c7487d7f7-xtvgn 10.244.2.81

The replacement has a new name and a new address, 10.244.2.81 instead of 10.244.2.80. Any client holding the old address is now broken. This is the problem in one screen of output.

Step 5 — Create the ClusterIP Service

ClusterIP is the default Service type and the one used for traffic between Pods inside the cluster. It is reachable only from within the cluster.

apiVersion: v1
kind: Service
metadata:
name: voucher-api
namespace: retail
labels:
app: voucher-api
spec:
type: ClusterIP
# The selector is what links this Service to Pods. It matches labels on the Pod,
# not on the Deployment, and it is the only thing that decides membership.
selector:
app: voucher-api
ports:
- name: http
port: 80 # the port this Service listens on
targetPort: http # the named port on the Pod, which is 8080
protocol: TCP
kubectl apply -f 3-voucher-api-clusterip.yaml
kubectl get svc voucher-api -n retail
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
voucher-api ClusterIP 10.96.129.205 <none> 80/TCP 33s

EXTERNAL-IP is <none>, which is the whole point: this address is internal.

Notice targetPort: http. Referring to the port by the name declared in the Pod template is better than repeating 8080, because changing the container's port then requires editing only one file.

Confirm the Service found the Pods:

kubectl get endpointslice -n retail -l kubernetes.io/service-name=voucher-api \
-o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\n"}{end}'
10.244.1.73
10.244.2.79
10.244.2.81

Three addresses, including 10.244.2.81, the replacement Pod from Step 4. The Service picked it up on its own.

Step 6 — Deploy a Debugging Pod

A cluster-internal DNS name cannot be resolved from your laptop, so testing has to happen from inside. Keep a Pod with networking tools around for this.

apiVersion: v1
kind: Pod
metadata:
name: netshoot
namespace: retail
labels:
app: netshoot
spec:
containers:
- name: netshoot
image: nicolaka/netshoot:latest
imagePullPolicy: IfNotPresent
command: ["sleep", "infinity"]
resources:
requests:
cpu: "25m"
memory: "64Mi"
limits:
cpu: "200m"
memory: "128Mi"
restartPolicy: Always
kubectl apply -f 4-netshoot-pod.yaml

The sleep infinity command is what keeps it alive. Without it the container would exit immediately and the Pod would restart in a loop.

Step 7 — Resolve the Service by Name

kubectl exec netshoot -n retail -- nslookup voucher-api.retail.svc.cluster.local
Name: voucher-api.retail.svc.cluster.local
Address: 10.96.129.205

That is the Service's virtual IP, not any Pod's address. The short form works too:

kubectl exec netshoot -n retail -- getent hosts voucher-api
10.96.129.205 voucher-api.retail.svc.cluster.local voucher-api

Step 8 — Watch Requests Spread Across Replicas

Send twelve requests through the Service and count which Pod answered each one:

kubectl exec netshoot -n retail -- sh -c 'for i in $(seq 1 12); do curl -s http://voucher-api/whoami; done' \
| sort | uniq -c | sort -rn
6 voucher-api-c7487d7f7-vszsl (10.244.1.73) on desktop-worker2
4 voucher-api-c7487d7f7-xtvgn (10.244.2.81) on desktop-worker
2 voucher-api-c7487d7f7-n8v28 (10.244.2.79) on desktop-worker

All three replicas received traffic through one stable name. Note that the split is 6-4-2, not 4-4-4. This cluster runs kube-proxy in iptables mode, which chooses a backend at random rather than in strict rotation. Over thousands of requests it evens out. If you expected exact round-robin and are counting a handful of requests, this is why the numbers look wrong.

Step 9 — What a Broken Selector Looks Like

This is the single most common Service mistake, so it is worth causing on purpose. Create a Service whose selector matches no Pod:

kubectl create service clusterip orphan-demo --tcp=80:8080 -n retail --dry-run=client -o yaml \
| sed 's/app: orphan-demo/app: does-not-exist/' | kubectl apply -f -

The Service exists and resolves, but it has no backends:

kubectl get endpointslice -n retail -l kubernetes.io/service-name=orphan-demo \
-o jsonpath='{.items[0].endpoints}'
null

And every request fails immediately:

kubectl exec netshoot -n retail -- curl -s -m 5 http://orphan-demo
connection refused (no endpoints)

Remember this shape. When a Service "does not work", check its endpoints first. An empty list almost always means the selector and the Pod labels disagree.

kubectl delete svc orphan-demo -n retail

Step 10 — A Note on kubectl expose

You can create a Pod and a Service together in one command:

kubectl run scratch-nginx --image nginx --expose=true --port=80
kubectl get pods scratch-nginx --show-labels
NAME READY STATUS LABELS
scratch-nginx 1/1 Running run=scratch-nginx

The generated Service gets the selector run=scratch-nginx, matching the label kubectl added automatically. It works, and it is fine for a five-second experiment.

Do not use it for anything real. It always produces a ClusterIP Service, it hides most of what a Service can do, and expressing anything beyond the defaults turns the command into something longer and less readable than the YAML would have been. Services are long-lived objects that belong in version control. Write them declaratively.

Step 11 — Clean Up

kubectl delete namespace retail
kubectl delete pod scratch-nginx
kubectl delete svc scratch-nginx

Summary

  1. Kubernetes solves four networking problems. Services solve Pod to Service.
  2. Every Pod gets a unique cluster-private IP from a CNI plugin. That address dies with the Pod.
  3. Never hardcode a Pod IP address. Recreating a Pod from the same manifest gives it a different one.
  4. A Service is a long-lived object with a stable virtual IP and a stable DNS name of the form <name>.<namespace>.svc.cluster.local.
  5. The link between a Service and its Pods is the selector matching Pod labels, and nothing else.
  6. ClusterIP is the default type and is reachable only from inside the cluster.
  7. Traffic distribution depends on the kube-proxy mode. iptables picks randomly; IPVS offers real algorithms and session affinity.
  8. When a Service seems broken, check its endpoints first. An empty list means the selector matches nothing.
  9. Use kubectl expose for experiments only. Write real Services as YAML.

In the next article we take the same idea outside the cluster, with NodePort, LoadBalancer, ExternalName, and headless Services.