HashiCorp Vault + Vault Secrets Operator on Kubernetes: End-to-End Guide

Table of Contents

1. Overview and Architecture
2. Prerequisites
3. Step 1: Install HashiCorp Vault with Helm
4. Step 2: Initialize and Unseal Vault
5. Step 3: Access the Vault Web UI
6. Step 4: Configure Vault via the Web UI
7. Step 5: Install the Vault Secrets Operator (VSO)
8. Step 6: Create the Kubernetes Custom Resources for reward-api
9. Step 7: Wire the Secret into the Application Deployment
10. Step 8: Verify the End-to-End Flow
11. Troubleshooting: Real Errors We Hit and How We Fixed Them
12. Summary

1. Overview and Architecture

The goal of this setup is simple: when an operator changes a secret in the Vault Web UI, the change must automatically propagate to the application pods — no manual kubectl work required after the initial setup.

The components involved:

1. HashiCorp Vault — runs inside the cluster in High Availability (HA) mode with 3 replicas using integrated Raft storage. The Web UI is exposed via a NodePort service.
2. Vault Secrets Operator (VSO) — a Kubernetes operator that watches Vault and synchronizes Vault secrets into native Kubernetes Secret objects. It can also trigger a rolling restart of Deployments when a secret changes.
3. The application (reward-api) — a Deployment named b-reward-api-dev in the reward-api namespace, which consumes all of its configuration as environment variables from a Kubernetes Secret.

The final data flow:

Note: because the application consumes secrets as environment variables, a pod restart is mandatory — environment variables are read once at process start and never refreshed. That is exactly why rolloutRestartTargets is used.

We deliberately disabled two optional Vault components, because VSO covers our use case entirely:

1. Vault Agent Injector — injects sidecar containers that write secrets as files into pods. Not needed (we do not want file-based secrets).
2. Vault CSI Provider — mounts secrets as filesystem volumes through the Secrets Store CSI Driver. Not needed for the same reason.

2. Prerequisites

1. A running Kubernetes cluster (in our case, nodes on Hetzner Cloud with the hcloud-volumes StorageClass).
2. kubectl configured against the cluster.
3. helm v3 installed.

Add the HashiCorp Helm repository:

3. Step 1: Install HashiCorp Vault with Helm

Create the namespace:

The Helm values file (values.yaml) used for the installation:

Install:

At this point the pods will show 0/1 Running and the logs will contain messages such as core: security barrier not initialized and seal configuration missing, not initialized. This is normal. Vault is installed but not yet initialized or unsealed.

4. Step 2: Initialize and Unseal Vault

Initialization and unsealing are manual by design — this is Vault’s security model (the master key is split into shares using Shamir’s Secret Sharing so that no single person can open the vault alone). It only needs to be repeated if the pods restart. For production, consider Auto-Unseal with an external KMS (azurekeyvault, awskms, gcpckms, or transit).

4.1 Initialize (only once, only on vault-0)

The output contains 5 Unseal Keys and the Initial Root Token. Store them somewhere safe — they cannot be recovered if lost.

4.2 Unseal vault-0 (with 3 different keys)

4.3 Join vault-1 and vault-2 to the Raft cluster, then unseal them

After unsealing, all pods should report 1/1 Running:

5. Step 3: Access the Vault Web UI

1. URL: http://<KUBERNETES_NODE_IP>:30820
2. Login method: select Token and paste the Root Token from the init step.

After logging in you will see the left navigation menu with: Dashboard, Secrets Engines, Access, Policies, Tools, and a Monitoring section with Raft Storage, Client Count, Seal Vault. All the configuration in the next step is done through this menu.

6. Step 4: Configure Vault via the Web UI

Everything in this section was done entirely in the browser — no CLI needed. We performed four tasks: create a read policy, enable the Kubernetes auth method, configure it, and create a role. Finally we enabled a KV secrets engine and created the application secret.

6.1 Create the read policy

1. In the left menu click Policies → Create ACL policy.
2. Name: vault-secrets-operator-policy
3. Policy body:

1. Click Save.

Important detail: the path in the policy must match the mount path of your KV engine. Our engine ended up mounted at kv/ (see section 6.4), so the policy grants read on kv/data/*. If your engine is mounted at secret/, the policy path would be secret/data/* instead. (KV v2 always inserts /data/ between the mount and the secret path in API terms.)

6.2 Enable the Kubernetes auth method

1. Left menu: Access → Auth Methods → Enable new method.
2. Select Kubernetes and click Enable Method (keep the default path kubernetes).

6.3 Configure the Kubernetes auth method

1. On the screen that opens after enabling, go to Configuration → Configure.
2. In the Kubernetes host field enter:

1. Save.

6.4 Enable the KV v2 secrets engine

A fresh Vault only ships with the cubbyhole/ engine (a per-token private scratch space that is created automatically, cannot be removed, and is not relevant for this setup). You must enable a KV engine yourself:

1. Left menu: Secrets Engines → Enable new engine +.
2. From the “Generic” list (KV, PKI Certificates, SSH, Transit, TOTP, LDAP, Kubernetes) select KV. KV is the classic key-value store for static secrets such as passwords, API keys and connection strings — exactly what an application config needs. Version 2 also keeps a version history of every secret.
3. Note the Path field. In our installation the engine was enabled with the path kv, so all secrets live under kv/. Keep this value in mind — it must match the mount: field of the VaultStaticSecret resource and the path in the ACL policy.
4. Click Enable Engine.

6.5 Create the application secret

1. Left menu: Secrets Engines → click kv/.
2. Click Create secret.
3. Path for this secret: rewardapi/config (do not type the mount prefix; it is added automatically; the value is case-sensitive).
4. Under Secret data add every environment variable your application needs as key/value pairs, for example:
5. key: veritabanisifresi, value: UnicoDevSifre123!
6. Click Save.

After saving, the secret’s Overview tab shows the canonical paths, which are very useful for debugging:

1. API path: /v1/kv/data/rewardapi/config
2. CLI path: -mount="kv" "rewardapi/config"

6.6 Create the role for the operator

1. Left menu: Access → Auth Methods → kubernetes → Create role.
2. Name: vault-secrets-operator-role
3. Bound service account names (type the value and press Add for each row):
4. vault-secrets-operator-controller-manager (the operator itself)
5. default (used by the VaultAuth resource in the application namespace)
6. Bound service account namespaces (press Add for each row):
7. hashicorp
8. reward-api (or * to allow every namespace in a dev cluster)
9. Expand the token options and set Generated Token’s Policies to vault-secrets-operator-policy (type it and press Add).
10. Optionally set Generated Token’s Initial TTL to 1h.
11. Leave all other fields (Alias name source, Audience, Bound CIDRs, Maximum TTL, Period, Token Type, etc.) at their defaults.
12. Click Save.

UI pitfall: in every “Add one item per row” field you must press the blue Add button after typing the value. If you press Save while the text is still only in the input box, the value is silently discarded.

7. Step 5: Install the Vault Secrets Operator (VSO)

The operator values file (operator-values.yaml):

Install and verify:

8. Step 6: Create the Kubernetes Custom Resources for reward-api

Two custom resources are needed per application namespace. These are Kubernetes resources (they do not appear anywhere in the Vault UI) and are applied once with kubectl. After that, day-to-day secret changes are done exclusively in the Vault UI.

1. VaultAuth — one per namespace; tells VSO how to authenticate to Vault. All applications in the same namespace share it.
2. VaultStaticSecret — one per application/secret; tells VSO which Vault path to sync into which Kubernetes Secret.

8.1 VaultAuth (apps/reward-api/vault-auth.yaml)

8.2 VaultStaticSecret (apps/reward-api/vault-static-secret.yaml)

8.3 Apply

Namespace rule: the VaultStaticSecret, the destination Secret, the referenced VaultAuth and the restarted Deployment must all live in the same namespace (reward-api here). For each additional application, copy these two files and adjust namespace, path, destination.name and rolloutRestartTargets.name. Only one VaultAuth is needed per namespace, no matter how many applications it contains.

9. Step 7: Wire the Secret into the Application Deployment

The application consumes all keys of the synced Secret as environment variables:

If the application previously used both a ConfigMap and a Secret, the recommended approach is to move both kinds of values into Vault (for example under rewardapi/config and rewardapi/secrets), sync them with two VaultStaticSecret resources, and reference both with envFrom. This gives you a single management surface: the Vault UI.

10. Step 8: Verify the End-to-End Flow

10.1 Verify the Kubernetes Secret was created

Note: VSO adds an extra key named _raw containing the whole payload — this is expected.

10.2 Inspect the environment variables inside the running pod

10.3 Test a live update from the UI

1. In the Vault UI open Secrets Engines → kv/ → rewardapi/config.
2. Click Create new version +, change a value, click Save.
3. Within 30 seconds (refreshAfter: 30s):
4. the Kubernetes Secret is updated, and
5. the Deployment b-reward-api-dev is rolling-restarted automatically.
6. Confirm:

KV v2 keeps a Version History, so you can also roll back to a previous version from the UI; the rollback propagates to the pods through exactly the same mechanism.

11. Troubleshooting: Real Errors We Hit and How We Fixed Them

Every one of the following errors occurred during this setup. They are listed in the order we encountered them.

11.1 unknown field "spec.mountPath" when applying VaultAuth

Cause: the VaultAuth CRD field is spec.mount, not spec.mountPath.

Fix: rename the field to mount.

11.2 unknown field "spec.refreshInterval" when applying VaultStaticSecret

Cause: the correct field name is refreshAfter.

Fix: rename refreshInterval to refreshAfter.

11.3 403 – service account name not authorized

Cause: the Vault role did not include the service account used by the VaultAuth resource (default in namespace reward-api).

Fix (UI): Access → Auth Methods → kubernetes → vault-secrets-operator-role → Edit: add default to Bound service account names and reward-api to Bound service account namespaces (remember to press Add for each entry), then Save. VSO retries automatically; no restart needed.

11.4 empty response from Vault, path="secret/data/rewardapi/config"

Cause: a mount-path mismatch. The VaultStaticSecret said mount: secret, but the KV engine was actually enabled at the path kv. The secret’s real API path (visible on its Overview tab in the UI) was /v1/kv/data/rewardapi/config.

Fix:

1. Change mount: secret to mount: kv in the VaultStaticSecret and re-apply.
2. Update the ACL policy path from secret/data/* to kv/data/* (Policies → edit in the UI).

Tip: when in doubt, always check the API path shown on the secret’s Overview page in the UI — it tells you the exact mount and path to use.

11.5 Pod error: secret "reward-api-config" not found

Cause: the pod referenced the Kubernetes Secret before VSO managed to create it (which it could not, due to errors 11.3 and 11.4).

Fix: resolve the sync errors above; as soon as VSO creates the Secret, the pod starts on its own. To speed it up: kubectl rollout restart deployment b-reward-api-dev -n reward-api.

11.6 General diagnostic commands

12. Summary

With this setup, the entire secret lifecycle is managed from the Vault Web UI: create or edit a value, press Save, and within 30 seconds the Kubernetes Secret is updated and the application pods are restarted with the new configuration — no kubectl, no CI/CD run, no manual restarts.