Integrating HashiCorp Vault Across Two Separate Kubernetes Clusters

Vault runs in one cluster, your applications run in another. Here is how to set up secret synchronization with the Vault Secrets Operator.

The Scenario

We have two independent Kubernetes clusters:

1. Cluster A — A self-managed k3s installation. HashiCorp Vault runs here, in the hashicorp namespace, exposed to the outside world through an ingress at https://vault.example.com.
2. Cluster B — A managed Kubernetes service (DigitalOcean Kubernetes / DOKS in this article). The applications live here and need to pull their secrets from Vault.

The goal: every application in Cluster B should be able to read its own secrets from Vault — and only its own. When a value changes in Vault, that change should propagate to the applications automatically.

The Part of the Architecture That Needs to Click

The most confusing aspect of this setup is this: the two clusters are not connected at the network level. There is no VPN, no peering, no service mesh. Pods cannot see each other. A svc.cluster.local address will not resolve in the other cluster.

Instead, there are two independent outbound HTTPS connections:

Direction 1: Application → Vault

A pod in Cluster B sends a request to https://vault.example.com. This is no different from calling any external API. From the pod's perspective, Vault is just a service on the internet.

Direction 2: Vault → Cluster B API Server

This direction is easy to overlook, and it is where the setup most often breaks.

With the Kubernetes auth method, a pod tells Vault "I am the mobile-api service account" and presents a JWT. Vault cannot simply take that JWT at face value — anyone can fabricate a JWT. It has to be verified.

Kubernetes provides the TokenReview API for exactly this: "here is a token, is it valid and who does it belong to?" Vault makes this call against Cluster B's API server. Which means Vault needs network access to Cluster B's API server.

On managed Kubernetes services the API server is publicly reachable by default (it is the address in your kubeconfig), so this usually works without any extra setup. But if outbound traffic is restricted in the environment where Vault runs, this direction will fail — in that case you need a method that does not require a reverse connection, such as AppRole.

The full flow

1. The pod sends its service account token to Vault
2. Vault calls TokenReview on Cluster B's API server using a "reviewer" identity
3. The API server confirms the token and reports which service account it belongs to
4. Vault returns its own token, carrying the relevant policies, to the pod
5. The pod reads its secret with that token

Prerequisites

1. A Vault instance running in Cluster A, initialized and unsealed
2. Vault reachable from outside with a valid certificate
3. Cluster B's API server reachable from Cluster A
4. A kubectl context for both clusters
5. Administrative access to Vault (enough to enable an auth method)

Step 1 — Verify Vault Is Reachable

Where: Any machine · Tool: curl

Look for these fields in the output:

You need sealed: false and standby: false. The fact that the command works without the -k flag also proves the certificate is publicly trusted — this will matter later when we configure skipTLSVerify.

Step 2 — Create the Reviewer ServiceAccount

Where: Cluster B · Tool: kubectl

For Vault to make TokenReview calls against Cluster B, it needs an identity in that cluster. This step creates it.

First, make sure you are on the right cluster:

Then apply the following manifest:

What each of the three objects does

ServiceAccount — Vault's identity inside Cluster B. Vault does not run a pod here, but it will talk to the API, so it needs an account.

ClusterRoleBinding — Grants that account the system:auth-delegator role. This is a built-in Kubernetes role with a very narrow scope: create on tokenreviews and subjectaccessreviews, nothing else. This account cannot list pods, read secrets, or see deployments. The name reflects its purpose — it exists so authentication can be delegated to an external system.

Secret — A long-lived JWT that proves the ServiceAccount's identity. Since Kubernetes 1.24, creating a ServiceAccount no longer generates a token secret automatically; pods get short-lived projected tokens instead. Vault is not a pod, so it needs an independent, non-expiring token. The combination of type: kubernetes.io/service-account-token and the annotation tells the token controller to generate one.

Verify the token was actually populated — this step can silently come up empty:

You should see a value greater than zero. If it comes back empty, wait a few seconds and try again.

Step 3 — Collect Cluster B's Details

Where: Cluster B · Tool: kubectl

No resources are created in this step — we only read three values:

These commands print nothing — one writes to a variable, the other to a file. Verify the results:

On macOS, some versions of base64 -d silently produce empty output. If the file is 0 bytes, use the uppercase flag:

These variables only live in the current shell session. Open a new tab and they are gone — and the command in the next step will silently send an empty value.

Step 4 — Test the Reverse Connection

Where: Cluster A · Tool: kubectl

Do not skip this step. It tells you upfront whether the whole approach will work.

If you get JSON back, Vault can reach Cluster B's API server. If it times out, something is blocking outbound traffic — and you should switch to AppRole instead of Kubernetes auth.

Step 5 — Configure the Auth Method in Vault

Where: Vault · Tool: vault CLI

Run these commands from the machine that holds cluster-b-ca.crt. If you run them from inside the Vault pod, the file referenced with @ will not be found and the CA field will end up empty.

What the parameters mean

The kubernetes-cluster-b path — Do not use the default kubernetes path. If the cluster hosting Vault already has a Kubernetes auth mount, you would overwrite it. Every cluster gets its own mount.

disable_local_ca_jwt=true — Critical. Without it, Vault tries to use the service account token and CA from its own pod (in Cluster A), and verification against Cluster B fails.

use_annotations_as_alias_metadata=false — When this is enabled, Vault tries to read the annotations of the ServiceAccount that is logging in. But system:auth-delegator grants no permission to read ServiceAccounts, so every login attempt errors out. Leave it off unless you actually need it.

The partial-update trap

When you write a single field to this endpoint, some of the other fields are preserved and some are reset. Writing only use_annotations_as_alias_metadata, for example, wipes token_reviewer_jwt. Always pass every field together on any vault write .../config call.

Verify:

Expected values:

1. kubernetes_host — Cluster B API server URL
2. kubernetes_ca_cert — PEM content (not n/a)
3. disable_local_ca_jwt — true
4. use_annotations_as_alias_metadata — false
5. token_reviewer_jwt_set — true

Step 6 — Templated Policy and a Shared Role

Where: Vault · Tool: vault CLI

Rather than writing a separate policy and role for every application, use a single templated policy. Vault substitutes the name of the service account that logged in directly into the policy path at request time.

First, get the mount's accessor:

You will get something like auth_kubernetes_a1b2c3d4. The policy references it:

Then a single role shared by all applications:

With this in place:

1. A pod logging in as mobile-api can only read secret/apps/mobile-api/*
2. A pod logging in as profile-api can only read secret/apps/profile-api/*

They cannot see each other's secrets, even when running in the same namespace.

Things to watch out for

bound_service_account_names='*' means every service account in Cluster B can log in to Vault. Access is constrained by the policy, so no pod can read another's secrets, but two conditions must hold:

1. Every application needs its own service account. Two services sharing the default service account will both see everything under secret/apps/default/*.
2. Never place secrets belonging to another service under secret/apps/<sa-name>/.

If your applications run in separate namespaces, including the namespace in the path gives tighter isolation:

A test secret

Step 7 — Install the Vault Secrets Operator

Where: Cluster B · Tool: helm

VSO is a controller that runs continuously inside Cluster B. It polls Vault for changes, updates local Kubernetes Secrets, and restarts pods when needed.

The most common mistake is leaving the in-cluster address in this file:

Leave skipTLSVerify set to false. Setting it to true keeps the traffic encrypted, but VSO stops verifying the identity of the server it connects to. An attacker in the path could then present a forged certificate, harvest the pod's service account token, and use that token to read secrets from the real Vault. If the curl in Step 1 worked without -k, your certificate is already valid and this setting is unnecessary.

Verify:

Step 8 — Application-Side Resources

Where: Cluster B · Tool: kubectl

The Vault side is now done. Each new application only needs these three objects:

Why there is no vaultConnectionRef

VaultConnection is a namespace-scoped resource, and the default connection created by Helm lives in the operator's namespace. When your VaultAuth sits in a different namespace, writing vaultConnectionRef: default makes VSO look for that resource in its own namespace, producing this error:

Leave the field out entirely and VSO falls back to the connection in the operator namespace. If you prefer to be explicit, give the full path:

The deployment side

Your application's deployment must name the service account, otherwise the pod runs as default and Vault sees the wrong identity:

How automatic refresh works

refreshAfter controls how often VSO polls Vault. When a value changes:

1. VSO notices the difference
2. It updates the local Kubernetes Secret
3. It restarts the deployments listed under rolloutRestartTargets
4. Pods come back up with the new value

Without rolloutRestartTargets, the Secret is updated but a pod reading it as environment variables keeps the old value in memory. If you use a volume mount and your application re-reads the file, no restart is needed.

60s is handy for testing, but in production it means one Vault request per service per minute. If your values rarely change, 300s or 600s is more reasonable.

Step 9 — Verification

Start with a manual login. This takes VSO out of the picture and shows you Vault's raw response:

You should see cluster-b-apps under policies and service_account_name: mobile-api under metadata. The latter is what makes the templated policy work.

Then check the VSO side:

You are looking for status.valid: true. To follow the logs:

To trigger a reconcile without waiting:

Common Errors

VaultConnection "default" not found

The vaultConnectionRef: default line in your VaultAuth was used in a namespace other than the operator's. Remove the line or give the fully qualified path.

Code: 403 — permission denied

Vault's most generic error, and it hides the real cause. Check these in order:

1. vault read auth/kubernetes-cluster-b/config — is kubernetes_ca_cert showing n/a? Without the CA, TokenReview cannot happen.
2. In the same output, is token_reviewer_jwt_set false? A partial update may have wiped it.
3. Is disable_local_ca_jwt set to true?
4. vault read auth/kubernetes-cluster-b/role/cluster-b-apps — does the role exist, and does bound_service_account_namespaces cover the namespace in question?
5. Do the audiences match? If the role has audience=vault, the VaultAuth needs audiences: [vault]. Having one without the other fails verification.

The real cause is usually visible in the Vault server log:

Code: 500 — failed to get service account ... is forbidden

This error is actually good news: TokenReview succeeded and the problem is in the step after it. use_annotations_as_alias_metadata is still enabled, and Vault is trying to reach a resource the reviewer has no permission for.

The fix is to turn the setting off. Alternatively, grant the reviewer the extra permission:

x509: certificate signed by unknown authority

Vault's certificate is not publicly trusted. The right fix is not skipTLSVerify: true but making the CA known to VSO:

HTML where JSON was expected

If Vault sits behind a CDN or WAF, bot protection may be returning a challenge page to VSO. VSO is a Go HTTP client and cannot solve JavaScript challenges. Exempt the Vault hostname from bot protection and rate limiting rules, and make sure /v1/* paths are never cached.

The command ran but produced no output

Assigning to a variable (VAR=$(...)) and redirecting to a file (> file) both print nothing. If there had been an error, you would have seen it. Confirm the results with echo and ls.

Maintenance and Growth

Adding a new application

You do not touch Vault's configuration. Just create the three objects in Cluster B (ServiceAccount, VaultAuth, VaultStaticSecret) and write the secret:

The secret path must match the service account name exactly — that is what the templated policy relies on.

The lifetime of the reviewer token

The token created in Step 2 never expires and is not rotated. If it is compromised, an attacker can perform TokenReview calls; that alone grants no ability to read secrets, but it is still a sensitive value stored in Vault.

Narrowing access

Since Vault is publicly exposed, consider restricting source IPs at the ingress. If Vault sits behind a CDN, every request arriving at the ingress appears to come from the CDN's IPs — so the restriction belongs on the CDN side, while the ingress should be configured to accept only the CDN's IP ranges.

Alternative: AppRole

If Vault cannot reach Cluster B's API server, Kubernetes auth is off the table. AppRole requires no reverse connection and is simpler to set up, but it does not give you per-pod identity — the secret_id lives in a Kubernetes Secret, and rotating it is a problem you have to solve separately.

Summary

No Kubernetes resources are created in the cluster where Vault runs. The only changes there are to Vault's own configuration, made through the vault CLI or the web UI.

The whole setup is a one-time effort. After that, onboarding a new application is three small manifests and a single vault kv put.