External Secret Operator Integration with Azure Key Vault

Introduction:

Kubernetes (K8s) has emerged as a popular container orchestration platform, and as organizations adopt K8s for their workloads, they must also consider robust secret management solutions. Azure Key Vault, a cloud-based secret management service provided by Microsoft Azure, is a powerful choice for securing sensitive data. Integrating Azure Key Vault with Kubernetes can enhance the security posture of your applications and simplify secret management.

One effective way to bridge the gap between Kubernetes and Azure Key Vault is by leveraging the External-Secret Operator.

In this blog post, we will explore the concept of secret management in Kubernetes, delve into the External-Secret Operator, and guide you through integrating it with Azure Key Vault.

The Challenge: Vault Downtime

Picture this scenario: You rely on HashiCorp Vault to store secrets for your applications, and everything is running smoothly. However, technology sometimes throws curveballs. If your Vault server experiences downtime, it can spell trouble. You may not be able to access your secrets when you need them most.

The Solution: Azure Key Vault ( in our case )

To ensure business continuity and data security, integrating your backup tool's (velero) secrets with Azure Key Vault is a smart move. Azure Key Vault is a robust and reliable cloud-based solution for secret storage. Even when your primary Vault system faces issues, Azure Key Vault remains accessible and you can restore your environment in a disaster recovery situation.

Step 1: Deploy the External Secrets Operator

helm repo add external-secrets https://charts.external-secrets.io  
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace

Step 2: Integration with Azure Key Vault

1. Set the name of the existing Azure Resource Group by setting the RESOURCE_GROUP variable as described below:

$ RESOURCE_GROUP=<Azure_Resource_Group>

2. To obtain the ID of the Azure tenant as it will be used later when integrating the External Secrets Operator and Azure:

$ TENANT_ID=$(az account show --query tenantId | tr -d \")

Note: Ensure you have az cli configured with auth

Step 3: Create and Store a Sensitive Asset

1. Create the Vault

$ VAULT_NAME=external-secret-vault   # unique and be between 3 and 24 characters in length  

$ az keyvault create --name $VAULT_NAME --resource-group $RESOURCE_GROUP

2. Add a secret to the newly created Key Vault

$ az keyvault secret set --name secret-name --vault-name $VAULT_NAME --value "Azure Key Vault"

Step 4: Create an Azure App Registration and Service Principal

Azure App Registration is used to communicate between the External Secrets Operator and Azure.

1. First, create a new App Registration called eso and extract the ID of the newly created app registration:

APP_ID=$(az ad app create --display-name "external-secret-operator" --query appId | tr -d \")

2. Create a Service Principal associated with the previously created App Registration

SERVICE_PRINCIPAL=$(az ad sp create --id $APP_ID --query id | tr -d \")

3. Add permissions to the App Registration to enable user impersonation between the External Secrets Operator and Azure Key Vault.

$ az ad app permission add --id $APP_ID --api-permissions <API_PERMISSION_ID>=<SCOPE> --api <API_ID>

Typically, the API value is the Application ID or Client ID of the Azure API you want to access. You can find this value in the Azure API's configuration.

The api-permission value represents the specific permissions or scopes you want to grant. The Azure API itself defines these permissions. You'll need to refer to the documentation of the Azure API to find the correct permissions (scopes) you want to grant.

Note: We did this manually on the Azure portal but it can be automated using Terraform, Crossplane or az cli

A. Exposing App API

B. Given API permission to APP

C Adding role assignment to Key vault

4. Create a password for the App Registration

APP_PASSWORD=$(az ad app credential reset - id $APP_ID - query password | tr -d \")

Step 5: Accessing Secrets Stored in Azure Key Vault From k8s

1. Create a Secret in the external-secret namespace containing the credentials to the Azure App Registration created in the prior section:

$ Kubectl create secret generic azure-secret-sp --from-literal=ClientID=$APP_ID --from-literal=ClientSecret=$APP_PASSWORD

2. Create ClusterSecretStore that specifies connecting to an external secrets manager.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: azure-secret-store
  namespace: external-secrets
spec:
  provider:
    azurekv:
      tenantId: "XXXXX-d7f9-XXXXXX"
      vaultUrl: https://XXXX..vault.azure.net/
      authSecretRef:
        clientId:
          name: azure-secret-sp
          key: ClientID
          namespace: external-secrets
        clientSecret:
          name: azure-secret-sp
          key: ClientSecret
          namespace: external-secrets

$ kubectl apply -f ClusterSecretStore.yaml

3. Finally, to synchronize the property stored within the Azure Key Vault create an ExternalSecret resource containing.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: external-secret
  namespace: app-namespace
spec:
  refreshInterval: "1h"
  secretStoreRef:
    kind: ClusterSecretStore
    name: azure-secret-store
  target:
    name: k8s-secret-name
    creationPolicy: Owner
  data:
    secretKey: cloud-credentials
    remoteRef:
      key: key-vault-secret-name

$ kubectl apply -f ExternalSecret.yaml

4. Validate external secret

kubectl get css 
kubectl get es

Final Steps :

Add secret as a volume created by the external secret operator.

# Mount it to container as a volume
apiVersion: v1
Kind: Pod
metadata:
  name: app-name
spec:
  Containers:
  - name: container-name
    image: docker-image-name
    volumeMounts:
      mountPath: /credentials
      name: cloud-credentials
  Volumes: 
    name: cloud-credentials
      secret:
        defaultMode: 420
        secretName: cloud-credentials //created by external secret operator


Kubectl apply -f pod.yaml

Conclusion:

External Secrets offers flexibility, enabling multi-tenancy by utilizing the same Azure Key Vault and leveraging namespace annotations to restrict access to specific secrets. This approach provides granular control and enhanced security for managing and accessing secrets within the cluster.

Thanks for reading this far! We appreciate your comments and feedback.

About The Author
Suraj Solanki
Senior DevOps Engineer
LinkedIn: https://www.linkedin.com/in/suraj-solanki