Managing Secrets in Kubernetes with External Secrets Operator

Kubernetes Secrets are not secure by default—they’re just base64 encoded. External Secrets Operator (ESO) syncs secrets from external providers into Kubernetes securely.

The Problem

# This "secret" is just base64 encoded
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  password: c2VjcmV0MTIz  # echo "secret123" | base64

Anyone with cluster access can decode it:

echo "c2VjcmV0MTIz" | base64 -d
# secret123

The Solution

Store secrets in a dedicated provider:

• AWS Secrets Manager
• HashiCorp Vault
• Google Secret Manager
• Azure Key Vault
• 1Password

ESO syncs them to Kubernetes:

External Provider  ──► ESO ──► k8s Secret ──► Pod
(encrypted at rest)     │         │
                       sync    mount

Installation

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace

AWS Secrets Manager

Create IAM Role

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:*:*:secret:*"
        }
    ]
}

Create SecretStore

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Create ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: database-secret
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: prod/database
        property: username
    - secretKey: password
      remoteRef:
        key: prod/database
        property: password

ESO creates the Kubernetes Secret automatically:

apiVersion: v1
kind: Secret
metadata:
  name: database-secret
data:
  username: <synced from AWS>
  password: <synced from AWS>

HashiCorp Vault

SecretStore for Vault

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: external-secrets-sa

ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secret
  data:
    - secretKey: api-key
      remoteRef:
        key: apps/myapp
        property: api_key

Google Secret Manager

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: gcp-secrets
spec:
  provider:
    gcpsm:
      projectID: my-project
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: my-cluster
          serviceAccountRef:
            name: external-secrets-sa

ClusterSecretStore

For cluster-wide access:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: company-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

Reference from any namespace:

spec:
  secretStoreRef:
    name: company-secrets
    kind: ClusterSecretStore

Advanced Patterns

Template Secrets

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: connection-string
spec:
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-connection
    template:
      data:
        connection_string: |
          postgresql://{{ .username }}:{{ .password }}@db.example.com:5432/app
  data:
    - secretKey: username
      remoteRef:
        key: prod/database
        property: username
    - secretKey: password
      remoteRef:
        key: prod/database
        property: password

Find by Tag

spec:
  dataFrom:
    - find:
        tags:
          environment: production
          team: platform

Refresh and Rotation

spec:
  refreshInterval: 5m  # Check for updates every 5 minutes

When secrets rotate in the provider, ESO syncs the changes.

GitOps Integration

ExternalSecrets are safe to commit to Git:

# This is safe - no actual secrets here
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
spec:
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  data:
    - secretKey: api-key
      remoteRef:
        key: apps/myapp/api-key

ArgoCD/Flux syncs the ExternalSecret, ESO creates the actual Secret.

Troubleshooting

Check Status

kubectl get externalsecret
# NAME         STORE          REFRESH INTERVAL   STATUS
# api-secret   aws-secrets    1h                 SecretSynced

kubectl describe externalsecret api-secret
# Shows sync status and errors

Common Issues

# Wrong secret path
SecretSyncedError: could not get secret data: resource not found

# Authentication failure  
SecretSyncedError: could not get secret data: access denied

# Malformed reference
InvalidSecretStore: missing configuration

Security Best Practices

1. Least Privilege

{
    "Effect": "Allow",
    "Action": ["secretsmanager:GetSecretValue"],
    "Resource": "arn:aws:secretsmanager:*:*:secret:prod/*"
}

Only allow access to needed secrets.

2. Namespace Isolation

Use SecretStore per namespace, not ClusterSecretStore everywhere.

3. Audit Logging

Enable audit logs in your secret provider:

# AWS CloudTrail
# Vault audit backend
# GCP audit logs

4. Rotation

spec:
  refreshInterval: 15m  # Frequent sync for rotating secrets

Final Thoughts

External Secrets Operator solves Kubernetes secret management properly:

• Secrets stay in dedicated, secure storage
• GitOps-friendly (no secrets in Git)
• Automatic synchronization
• Works with any major provider

If you’re running Kubernetes in production, ESO is essential.

Secrets belong in secret managers, not in Git.