External Secrets Operator Integration

Arcan provides an ESO-compatible webhook endpoint that lets the External Secrets Operator sync secrets from Arcan into Kubernetes Secrets automatically.

Why Webhook Provider?

ESO supports many providers natively (AWS SM, Vault, GCP SM), but Arcan uses the webhook provider -- a generic HTTP-based backend. This means:

No custom ESO controller or CRD required
Works with any ESO installation out of the box
Arcan controls the API contract (/api/v1/eso/...)

Prerequisites

External Secrets Operator installed in your cluster (install guide)
Arcan server running and reachable from the cluster (e.g., https://arcan.internal:8443)

Arcan API token created:

arcan token create --name="eso-operator" --scopes read

Quick Start

Step 1: Create the token Secret

kubectl create secret generic arcan-credentials \
  --namespace=default \
  --from-literal=token=arc_YOUR_API_TOKEN

Step 2: Trust Arcan's TLS certificate

If Arcan uses a self-signed or private CA certificate:

kubectl create configmap arcan-ca \
  --namespace=default \
  --from-file=ca.crt=/path/to/arcan-ca.crt

Skip this if Arcan uses a publicly trusted certificate (and remove the caProvider block from the manifests).

Step 3: Generate and apply manifests

arcan generate eso --realm=production --env=prod --namespace=default | kubectl apply -f -

Or use the static manifests in deploy/kubernetes/eso/ after replacing the REPLACE_ME placeholders:

kubectl apply -f deploy/kubernetes/eso/arcan-token-secret.yaml
kubectl apply -f deploy/kubernetes/eso/secret-store.yaml
kubectl apply -f deploy/kubernetes/eso/external-secret.yaml

API Endpoints

Endpoint	Method	Response	Use Case
`/api/v1/eso/{realm}/{key}?env=prod`	GET	`{"value": "secret-value"}`	Single secret
`/api/v1/eso/{realm}?env=prod`	GET	`{"data": {"KEY1": "val1", ...}}`	All secrets in realm

Both endpoints require Authorization: Bearer arc_... header.

Bulk Secrets

To sync all secrets from a realm at once, use the bulk endpoint with a separate SecretStore:

kubectl apply -f deploy/kubernetes/eso/external-secret-bulk.yaml

The bulk SecretStore points to /api/v1/eso/{realm}?env={env} (no key in the path) and uses $.data as the jsonPath to extract all key-value pairs.

ClusterSecretStore

For cross-namespace access, use a ClusterSecretStore:

arcan generate eso --realm=production --env=prod --cluster-wide | kubectl apply -f -

Or apply the static manifest:

kubectl apply -f deploy/kubernetes/eso/cluster-secret-store.yaml

With a ClusterSecretStore, ExternalSecrets in any namespace can reference kind: ClusterSecretStore in their secretStoreRef. The token Secret and CA ConfigMap must exist in the ESO controller namespace (typically external-secrets).

TLS Trust

Arcan requires TLS in all deployment modes. If using a self-signed certificate (the default for standalone mode), ESO needs to trust Arcan's CA.

The caProvider block in the SecretStore tells ESO where to find the CA:

caProvider:
  type: ConfigMap
  name: arcan-ca
  key: ca.crt

To extract Arcan's CA certificate:

# Run `arcan doctor` to find the CA certificate path, then copy it:
cp <ca-cert-path> ./arcan-ca.crt

# Or fetch it from a running server
openssl s_client -connect arcan.internal:8443 -showcerts </dev/null 2>/dev/null \
  | openssl x509 -outform PEM > arcan-ca.crt

Refresh Intervals

The refreshInterval on an ExternalSecret controls how often ESO polls Arcan:

Interval	Propagation delay	API load
`15m`	Up to 15 minutes	Low
`1h` (default)	Up to 1 hour	Very low
`5m`	Up to 5 minutes	Moderate
`1m`	Up to 1 minute	High

Choose based on how quickly secret changes need to propagate. For most workloads, 1h is sufficient.

Troubleshooting

ExternalSecret stuck in "SecretSyncedError"

Check the ESO controller logs:

kubectl logs -n external-secrets deploy/external-secrets -f

Common causes:

Connection refused: Arcan not reachable. Verify the URL and that the Arcan service exists in the cluster.
TLS handshake failure: CA certificate not trusted. Ensure the arcan-ca ConfigMap exists and contains the correct certificate.
401 Unauthorized: Invalid or expired token. Verify the token in the arcan-credentials Secret.
404 Not Found: Wrong realm slug or key name. Check the realm exists with arcan realm list.

Verify connectivity from the cluster

kubectl run arcan-test --rm -it --image=curlimages/curl -- \
  curl -sk -H "Authorization: Bearer arc_YOUR_TOKEN" \
  https://arcan.internal:8443/api/v1/eso/production/DATABASE_URL?env=prod

Check SecretStore status

kubectl get secretstore arcan -o yaml
kubectl describe secretstore arcan

A healthy SecretStore shows status.conditions with type: Ready and status: "True".

Check ExternalSecret sync status

kubectl get externalsecret my-app-secrets -o yaml
kubectl describe externalsecret my-app-secrets

Look at status.conditions and status.syncedResourceVersion to confirm secrets are syncing.

Complete End-to-End Example

Walk through the full flow from creating an Arcan token to verifying the Kubernetes Secret exists.

1. Create an Arcan token for ESO

arcan token create --name="k8s-eso-operator" --scopes read

# Output:
# Token: arc_eso_a1b2c3d4e5f6g7h8i9j0
# Scopes: read
# Save this token — it won't be shown again.

2. Store the token in Kubernetes

kubectl create secret generic arcan-credentials \
  --namespace=default \
  --from-literal=token=arc_eso_a1b2c3d4e5f6g7h8i9j0

3. Apply the SecretStore

# arcan-secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: arcan
  namespace: default
spec:
  provider:
    webhook:
      url: "https://arcan.internal:8443/api/v1/eso/production/{{ .remoteRef.key }}?env=prod"
      headers:
        Authorization: "Bearer {{ .authRef.SecretAccessKey }}"
        Content-Type: application/json
      result:
        jsonPath: "$.value"
      secrets:
        - name: SecretAccessKey
          secretRef:
            name: arcan-credentials
            key: token
      caProvider:
        type: ConfigMap
        name: arcan-ca
        key: ca.crt

kubectl apply -f arcan-secret-store.yaml

4. Apply the ExternalSecret

# app-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
  namespace: default
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: arcan
    kind: SecretStore
  target:
    name: myapp-secrets        # Name of the K8s Secret that will be created
    creationPolicy: Owner      # ESO owns and manages this Secret
  data:
    - secretKey: DATABASE_URL  # Key in the K8s Secret
      remoteRef:
        key: DATABASE_URL      # Key in Arcan
    - secretKey: REDIS_URL
      remoteRef:
        key: REDIS_URL
    - secretKey: STRIPE_SECRET
      remoteRef:
        key: STRIPE_SECRET

kubectl apply -f app-secrets.yaml

5. Verify the Kubernetes Secret was created

# Check ExternalSecret status
kubectl get externalsecret myapp-secrets
# NAME             STORE   REFRESH INTERVAL   STATUS
# myapp-secrets    arcan   15m                SecretSynced

# Verify the K8s Secret exists
kubectl get secret myapp-secrets
# NAME             TYPE     DATA   AGE
# myapp-secrets    Opaque   3      12s

# Confirm the keys are present (values are base64 encoded)
kubectl get secret myapp-secrets -o jsonpath='{.data}' | jq .
# {
#   "DATABASE_URL": "cG9zdGdyZXM6Ly91c2VyOnBhc3NAZGIuZXhhbXBsZS5jb206NTQzMi9teWFwcA==",
#   "REDIS_URL": "cmVkaXM6Ly9jYWNoZS5leGFtcGxlLmNvbTo2Mzc5",
#   "STRIPE_SECRET": "c2tfbGl2ZV9hYmMxMjNkZWY0NTY="
# }

6. Use the Secret in a Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: myapp
          image: myapp:latest
          envFrom:
            - secretRef:
                name: myapp-secrets  # All keys injected as env vars

Syncing All Secrets from a Realm (dataFrom)

Instead of listing each secret individually, you can sync every secret in a realm at once using the bulk endpoint and dataFrom:

# bulk-secrets.yaml
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: arcan-bulk
  namespace: default
spec:
  provider:
    webhook:
      url: "https://arcan.internal:8443/api/v1/eso/production?env=prod"
      headers:
        Authorization: "Bearer {{ .authRef.SecretAccessKey }}"
        Content-Type: application/json
      result:
        jsonPath: "$.data"
      secrets:
        - name: SecretAccessKey
          secretRef:
            name: arcan-credentials
            key: token
      caProvider:
        type: ConfigMap
        name: arcan-ca
        key: ca.crt
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-all-secrets
  namespace: default
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: arcan-bulk
    kind: SecretStore
  target:
    name: myapp-all-secrets
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: "all"  # The bulk endpoint returns all keys

kubectl apply -f bulk-secrets.yaml

# All secrets from the production realm are now in a single K8s Secret
kubectl get secret myapp-all-secrets -o jsonpath='{.data}' | jq 'keys'
# ["API_KEY", "DATABASE_URL", "REDIS_URL", "SENTRY_DSN", "STRIPE_SECRET"]

tip

Use dataFrom for microservices that need every secret in a realm. Use individual data entries when you want to cherry-pick specific secrets or map Arcan keys to different Kubernetes Secret keys.

Quick Reference: End-to-End in 5 Commands

If you want the fastest path from zero to a working Kubernetes Secret synced from Arcan:

# 1. Create a read-only token in Arcan
arcan token create --name="k8s-eso" --scopes read
# Output: arc_eso_a1b2c3d4e5f6g7h8i9j0

# 2. Store the token as a Kubernetes Secret
kubectl create secret generic arcan-credentials \
  --namespace=default \
  --from-literal=token=arc_eso_a1b2c3d4e5f6g7h8i9j0

# 3. Store the Arcan CA certificate (skip if using a publicly trusted cert)
kubectl create configmap arcan-ca \
  --namespace=default \
  --from-file=ca.crt=<ca-cert-path>

# 4. Generate and apply SecretStore + ExternalSecret manifests
arcan generate eso --realm=production --env=prod --namespace=default | kubectl apply -f -

# 5. Verify the Kubernetes Secret was created
kubectl get secret myapp-secrets -o jsonpath='{.data}' | jq 'keys'
# ["API_KEY", "DATABASE_URL", "REDIS_URL", "SENTRY_DSN", "STRIPE_SECRET"]

That is it. ESO will now poll Arcan every hour (default) and keep the Kubernetes Secret in sync. To use the secret in a pod:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp
          image: myapp:latest
          envFrom:
            - secretRef:
                name: myapp-secrets

Common ESO Status Conditions

When debugging, check kubectl describe externalsecret <name>. Here are the status conditions and what they mean:

Condition	Status	Meaning	Fix
`Ready`	`True`	Secrets are synced and up to date	Everything is working
`Ready`	`False`, reason `SecretSyncedError`	ESO cannot fetch from Arcan	Check connectivity, token, and realm name
`Ready`	`False`, reason `SecretStoreNotReady`	The referenced SecretStore is not healthy	Run `kubectl describe secretstore arcan`
`Ready`	`False`, reason `SecretDeleted`	The target K8s Secret was manually deleted	ESO will recreate it on the next refresh cycle
`Ready`	`False`, reason `InvalidStoreRef`	The SecretStore name or kind is wrong	Verify `secretStoreRef` matches your SecretStore
`Ready`	`False`, reason `ProviderError`	The webhook returned a non-200 status	Check Arcan server logs and token validity

To see the exact error message:

kubectl get externalsecret myapp-secrets -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'

Why Webhook Provider?​

Prerequisites​

Quick Start​

Step 1: Create the token Secret​

Step 2: Trust Arcan's TLS certificate​

Step 3: Generate and apply manifests​

API Endpoints​

Bulk Secrets​

ClusterSecretStore​

TLS Trust​

Refresh Intervals​

Troubleshooting​

ExternalSecret stuck in "SecretSyncedError"​

Verify connectivity from the cluster​

Check SecretStore status​

Check ExternalSecret sync status​

Complete End-to-End Example​

1. Create an Arcan token for ESO​

2. Store the token in Kubernetes​

3. Apply the SecretStore​

4. Apply the ExternalSecret​

5. Verify the Kubernetes Secret was created​

6. Use the Secret in a Deployment​

Syncing All Secrets from a Realm (dataFrom)​

Quick Reference: End-to-End in 5 Commands​

Common ESO Status Conditions​