Secrets Management — HashiCorp Vault, SOPS, and Sealed Secrets

You find a database password hardcoded in a Python file. Someone committed an AWS access key to a public GitHub repository three months ago, and it has been scraped by bots ever since. The production .env file is shared via Slack DM. These are not hypothetical scenarios — they happen every day, at companies of every size. In 2023, GitGuardian detected over 12 million hardcoded secrets in public GitHub commits. Secrets management is not optional. It is a fundamental requirement for any serious engineering team.

Why Secrets Management Matters

Secrets are any credentials your applications need to function — database passwords, API keys, TLS certificates, SSH keys, OAuth tokens, encryption keys. They are the keys to your kingdom, and mismanaging them leads to breaches.

Common ways secrets leak:
  ┌─────────────────────────────┐
  │ Hardcoded in source code    │──▶ Git history preserves them forever
  │ .env files in Git           │──▶ Anyone with repo access has the keys
  │ Shared via Slack/email      │──▶ Stored in plaintext in message logs
  │ CI/CD logs                  │──▶ Printed in build output by accident
  │ Docker image layers         │──▶ Anyone who pulls the image can extract
  │ Config files on servers     │──▶ Wide file permissions expose them
  └─────────────────────────────┘

The 12-factor app methodology says it clearly: store config in the environment, never in code. But environment variables are just the beginning. Real secrets management means encryption at rest, access control, audit logging, and automatic rotation.

HashiCorp Vault

Vault is the industry standard for secrets management. It provides encrypted storage, dynamic secret generation, fine-grained access control, and detailed audit logs.

Architecture

  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
  │ Application  │    │ CI/CD        │    │ Developer    │
  │ (AppRole)    │    │ (JWT auth)   │    │ (LDAP/OIDC)  │
  └──────┬───────┘    └──────┬───────┘    └──────┬───────┘
         │                   │                   │
         ▼                   ▼                   ▼
  ┌──────────────────────────────────────────────────────┐
  │                   HashiCorp Vault                     │
  │  ┌────────────┐  ┌────────────┐  ┌────────────────┐  │
  │  │ Auth       │  │ Secrets    │  │ Audit          │  │
  │  │ Methods    │  │ Engines    │  │ Logging        │  │
  │  │            │  │            │  │                │  │
  │  │ - AppRole  │  │ - KV v2    │  │ - File         │  │
  │  │ - JWT      │  │ - Database │  │ - Syslog       │  │
  │  │ - LDAP     │  │ - AWS      │  │ - Socket       │  │
  │  │ - K8s      │  │ - PKI      │  │                │  │
  │  └────────────┘  └────────────┘  └────────────────┘  │
  │                                                       │
  │  Storage Backend: Consul, Raft, S3, etc.             │
  └──────────────────────────────────────────────────────┘

Getting Started with Vault

# Run Vault in development mode (NOT for production)
docker run -d --name vault \
  -p 8200:8200 \
  -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' \
  -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' \
  hashicorp/vault:1.17

# Configure CLI
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='myroot'

# Verify connection
vault status

KV Secrets Engine

The KV (Key-Value) engine stores static secrets with versioning:

# Enable KV v2 secrets engine
vault secrets enable -path=secret kv-v2

# Store a secret
vault kv put secret/myapp/database \
  username="app_user" \
  password="s3cur3P@ssw0rd" \
  host="db-primary.internal" \
  port="5432"

# Read a secret
vault kv get secret/myapp/database

# Read a specific field
vault kv get -field=password secret/myapp/database

# Read as JSON (useful in scripts)
vault kv get -format=json secret/myapp/database | jq -r '.data.data.password'

# List all secrets at a path
vault kv list secret/myapp/

# Update a secret (creates a new version)
vault kv put secret/myapp/database \
  username="app_user" \
  password="n3wP@ssw0rd!" \
  host="db-primary.internal" \
  port="5432"

# Read a previous version
vault kv get -version=1 secret/myapp/database

# Delete a secret (soft delete — can be recovered)
vault kv delete secret/myapp/database

# Permanently destroy a version
vault kv destroy -versions=1 secret/myapp/database

Dynamic Secrets

Dynamic secrets are generated on demand and automatically revoked after a TTL. No more shared, long-lived passwords:

# Enable the database secrets engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/mydb \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db-primary.internal:5432/mydb" \
  allowed_roles="readonly,readwrite" \
  username="vault_admin" \
  password="vault_admin_password"

# Create a role that generates read-only credentials
vault write database/roles/readonly \
  db_name=mydb \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate temporary database credentials
vault read database/creds/readonly
# Key                Value
# ---                -----
# lease_id           database/creds/readonly/abc123
# lease_duration     1h
# username           v-approle-readonly-xyz789
# password           A1b2C3d4E5f6G7h8

# The credentials are automatically revoked after 1 hour
# No credential sharing, no long-lived passwords, full audit trail

AppRole Authentication

For applications and CI/CD pipelines, use AppRole authentication:

# Enable AppRole auth method
vault auth enable approle

# Create a policy for the application
vault policy write myapp-policy - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
path "database/creds/readonly" {
  capabilities = ["read"]
}
EOF

# Create an AppRole
vault write auth/approle/role/myapp \
  token_policies="myapp-policy" \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_ttl=720h

# Get the Role ID (stable, like a username)
vault read auth/approle/role/myapp/role-id

# Generate a Secret ID (rotatable, like a password)
vault write -f auth/approle/role/myapp/secret-id

# Application authenticates with Role ID + Secret ID
vault write auth/approle/login \
  role_id="abc-123-def" \
  secret_id="xyz-789-uvw"

# Returns a token the application uses to read secrets

Mozilla SOPS

SOPS (Secrets OPerationS) encrypts files in-place, leaving keys visible but values encrypted. This makes diffs readable and Git-friendly. SOPS supports age, PGP, AWS KMS, GCP KMS, and Azure Key Vault for encryption.

# Install SOPS
brew install sops

# Install age (modern encryption tool, recommended over PGP)
brew install age

# Generate an age key pair
age-keygen -o keys.txt
# Public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
# Store the private key securely — this decrypts your secrets

SOPS Configuration

# .sops.yaml — place in the root of your repository
creation_rules:
  # Encrypt production secrets with age
  - path_regex: environments/production/.*\.enc\.yaml$
    age: "age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

  # Encrypt staging secrets with a different key
  - path_regex: environments/staging/.*\.enc\.yaml$
    age: "age1yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy"

  # Encrypt all other secret files
  - path_regex: .*\.enc\.yaml$
    age: "age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Using SOPS

# Create a secrets file
cat > environments/production/secrets.enc.yaml << 'EOF'
database:
  username: app_user
  password: s3cur3P@ssw0rd
  host: db-primary.internal
redis:
  password: r3d1sP@ss
api_keys:
  stripe: sk_live_abc123
  sendgrid: SG.xyz789
EOF

# Encrypt the file in place
sops --encrypt --in-place environments/production/secrets.enc.yaml

# The file now looks like this (keys visible, values encrypted):
# database:
#     username: ENC[AES256_GCM,data:abc123...,type:str]
#     password: ENC[AES256_GCM,data:def456...,type:str]
#     host: ENC[AES256_GCM,data:ghi789...,type:str]

# Edit encrypted file (opens in $EDITOR, decrypted temporarily)
sops environments/production/secrets.enc.yaml

# Decrypt and print to stdout
sops --decrypt environments/production/secrets.enc.yaml

# Decrypt a specific key
sops --decrypt --extract '["database"]["password"]' environments/production/secrets.enc.yaml

# Git integration — safe to commit encrypted files
git add environments/production/secrets.enc.yaml
git commit -m "Update production database credentials"

SOPS with Git Hooks

# .githooks/pre-commit — prevent committing unencrypted secrets
#!/bin/bash

for file in $(git diff --cached --name-only | grep '\.enc\.yaml$'); do
  if ! grep -q "ENC\[AES256_GCM" "$file"; then
    echo "ERROR: $file appears to be unencrypted!"
    echo "Run: sops --encrypt --in-place $file"
    exit 1
  fi
done

Kubernetes Sealed Secrets

Sealed Secrets lets you encrypt Kubernetes Secret resources so they can be safely stored in Git. Only the Sealed Secrets controller running in your cluster can decrypt them.

# Install the Sealed Secrets controller in the cluster
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml

# Install kubeseal CLI
brew install kubeseal

# Verify the controller is running
kubectl get pods -n kube-system -l name=sealed-secrets-controller

Creating Sealed Secrets

# Step 1: Create a regular Kubernetes Secret (do NOT apply it)
kubectl create secret generic db-credentials \
  --from-literal=username=app_user \
  --from-literal=password=s3cur3P@ssw0rd \
  --namespace=production \
  --dry-run=client -o yaml > secret.yaml

# Step 2: Seal the secret
kubeseal --format yaml < secret.yaml > sealed-secret.yaml

# Step 3: Commit the sealed secret to Git (safe!)
git add sealed-secret.yaml
git commit -m "Add sealed database credentials"

# Step 4: Apply the sealed secret to the cluster
kubectl apply -f sealed-secret.yaml

# The controller decrypts it and creates a regular Secret
kubectl get secret db-credentials -n production

# sealed-secret.yaml — safe to store in Git
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    username: AgBz8...long-encrypted-string...
    password: AgCx9...long-encrypted-string...
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

Comparison Table

Feature	HashiCorp Vault	Mozilla SOPS	Sealed Secrets	Cloud Native (AWS SM, Azure KV)
Type	Centralized secrets server	File encryption tool	Kubernetes controller	Managed cloud service
Setup Complexity	High	Low	Medium	Low
Dynamic Secrets	Yes	No	No	Limited
Secret Rotation	Built-in	Manual	Manual	Built-in
Audit Logging	Comprehensive	Git history	Kubernetes events	Cloud audit logs
Access Control	Fine-grained policies	Encryption keys	Namespace-scoped	IAM policies
Git-Friendly	No (secrets not in Git)	Yes (encrypted files in Git)	Yes (sealed resources in Git)	No (external storage)
Kubernetes Native	Via injector/CSI	Via Flux decryption	Yes	Via CSI driver
Cost	Free (OSS) / Paid (Enterprise)	Free	Free	Per-secret pricing
Best For	Enterprise, dynamic secrets, multi-cloud	Small teams, GitOps, simple needs	Kubernetes-native GitOps	Single-cloud workloads

CI/CD Secrets Injection

Never store secrets in your CI/CD configuration. Use your platform's secrets management:

# GitHub Actions — using GitHub Secrets
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: |
          # Secrets are available as environment variables
          # They are masked in logs automatically
          ./deploy.sh

# GitHub Actions — using Vault
      - name: Retrieve secrets from Vault
        uses: hashicorp/vault-action@v3
        with:
          url: https://vault.company.com
          method: jwt
          role: github-actions
          secrets: |
            secret/data/myapp/database password | DB_PASSWORD ;
            secret/data/myapp/api-keys stripe | STRIPE_KEY

      - name: Deploy
        run: |
          echo "Database password is masked: $DB_PASSWORD"
          ./deploy.sh

Secret Rotation

Secrets should be rotated regularly. The best rotation is the kind you never have to think about:

# Vault handles rotation automatically with dynamic secrets
# For static secrets, use a rotation script:

#!/bin/bash
# rotate-db-password.sh

# Generate new password
NEW_PASSWORD=$(openssl rand -base64 32)

# Update the database user password
psql -h db-primary.internal -U admin -c \
  "ALTER USER app_user PASSWORD '${NEW_PASSWORD}';"

# Update Vault with the new password
vault kv put secret/myapp/database \
  username="app_user" \
  password="${NEW_PASSWORD}" \
  host="db-primary.internal" \
  port="5432"

# Trigger rolling restart of application pods
# (they will fetch the new secret on startup)
kubectl rollout restart deployment/myapp -n production

echo "Password rotated successfully at $(date)"

The 12-Factor App Secrets Pattern

# The correct way to consume secrets in your application:

# 1. Secrets are injected as environment variables at runtime
# 2. The application reads them from the environment
# 3. The application NEVER logs, prints, or exposes secret values
# 4. The application NEVER has fallback/default secret values

# Example: Docker Compose with Vault Agent
# docker-compose.yml
# services:
#   myapp:
#     image: myapp:v1.0.0
#     environment:
#       DATABASE_URL: "postgresql://${DB_USER}:${DB_PASS}@db:5432/mydb"
#     env_file:
#       - .env   # NEVER commit this file — add to .gitignore

# .gitignore — ALWAYS include these
echo ".env" >> .gitignore
echo ".env.*" >> .gitignore
echo "*.pem" >> .gitignore
echo "*.key" >> .gitignore
echo "*credentials*" >> .gitignore
echo "*secret*" >> .gitignore

Start with the simplest tool that meets your needs. If you are a small team using Kubernetes and GitOps, Sealed Secrets or SOPS will get you 90% of the way there. If you need dynamic secrets, fine-grained access control, and multi-cloud support, invest in Vault. But whatever you choose, the most important step is the first one — stop hardcoding credentials today.

---

Your secrets are encrypted, rotated, and properly managed. In the next post, we will explore platform engineering — building Internal Developer Platforms that give your developers self-service access to infrastructure without filing tickets.