Hashicorp Vault Multi Node Cluster Setup on AWS EC2 with DynamoDB

6 min readJan 7, 2023

Today, companies are moving their workloads to the public clouds for scalable and cost-effective solutions. However, this migration means they are leaving their on-premises security perimeters behind And in the cloud platform, you can no longer trust any request on the network even if it appears to be coming from your own infrastructure because your infrastructure is ephemeral, and can change at any time.

A similar use case we have solved in our environment where we have migrated our secrets from the parameter store to Vault.

What will we cover :

This will cover a multi-node vault cluster on AWS EC2 with the DynamoDB backend Storage. In production, there is most likely a Vault cluster using an AWS Auto Scaling group with an AWS load balancer and Route53 for a private custom vault endpoint.

So this guide will cover

1. Prerequisite for vault cluster
2. Installation and Configuration for Vault
3. Vault Initialization process
4. Demonstrate Vault logging

Prerequisite for Vault cluster:

1. DynamoDB Table
2. KMS Key
3. EC2 Instance (3 Minimum)
4. Allow 8200 and 8201 Firewall Port

Note: I haven’t included VPC configuration, security groups, and other resources here as this post is about setting up Vault production-grade setup. Assuming you already have a basic understanding of AWS and Linux. You can use any automation tool and manual process to set up these prerequisites.

Required Permissions to EC2:

The governing policy for the EC2 instance profile that Vault uses to access DynamoDB must contain the following permissions for Vault to perform the required operations on the DynamoDB table and KMS key :

  "Statement": [
    {
      "Action": [
        "dynamodb:DescribeLimits",
        "dynamodb:DescribeTimeToLive",
        "dynamodb:ListTagsOfResource",
        "dynamodb:DescribeReservedCapacityOfferings",
        "dynamodb:DescribeReservedCapacity",
        "dynamodb:ListTables",
        "dynamodb:BatchGetItem",
        "dynamodb:BatchWriteItem",
        "dynamodb:CreateTable",
        "dynamodb:DeleteItem",
        "dynamodb:GetItem",
        "dynamodb:GetRecords",
        "dynamodb:PutItem",
        "dynamodb:Query",
        "dynamodb:UpdateItem",
        "dynamodb:Scan",
        "dynamodb:DescribeTable"
      ],
      "Effect": "Allow",
      "Resource": [ "arn:aws:dynamodb:region:... dynamodb table ARN" ]
    },
----------------------------------------------------------------------
"Statement": [
    {
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Effect": "Allow",
      "Resource": [ "arn:aws:kmskeyid.. KMS Key ARN" ]
    },

Installation and Configuration for Vault :

Step 1: Pre-Vault Process

Create a vault user and group: 

$ groupadd --force --system vault
$ adduser --system --gid vault --no-create-home --comment "vault owner" --shell /bin/false vault >/dev/null

# A default profile for local vault address

cat << EOF > /etc/profile.d/vault.sh
export VAULT_ADDR=http://127.0.0.1:8200
EOF

Step 2: Download Vault Binary

$ cd /opt/ && sudo curl -o vault.zip  https://releases.hashicorp.com/vault/1.12.1/vault_1.12.1_linux_amd64.zip
$ sudo unzip vault.zip
$ sudo cp vault /usr/local/bin/
$ vault --version
$ vault -autocomplete-install

Step 3: Make Vault Configuration File

Vault requires a config file, so here we create that config file and set ownership and permissions.

$ mkdir /etc/vault.d 

$ cat > /etc/vault.d/vault.hcl <<-EOF

storage "dynamodb" {
  ha_enabled = "true"
  region     = "region"
  table      = "Table_Name"
}

seal "awskms" {
  region     = "region"
  kms_key_id = "KMS_KEY_ID"
}

listener "tcp" {
 address = "0.0.0.0:8200"
 cluster_address = "0.0.0.0:8201"
 tls_disable = true
}

api_addr = "http://custom-vault-internal-url.com or LB_URL"
cluster_addr = "http://instance-private-ip:8201"
cluster_name = "vault-cluster"
ui = true
log_level = "ERROR"
disable_mlock = true

EOF

Set Permission and Ownership to vault system user

$ chown -R vault:vault /etc/vault.d
$ chmod -R 0644 /etc/vault.d/*
$ mkdir /var/log/vault.d
$ chown vault:vault /var/log/vault.d/*

Step 4: Setup Vault Service

Enable systemd on Amazon Linux for Vault

$ cat > /etc/systemd/system/vault.service <<-EOF

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

EOF

Start Vault Service :

Reload systemd and enable vault

$ systemctl daemon-reload
$ systemctl start vault.service
$ systemctl enable vault.service
$ systemctl status vault.service 
$ ln -s /etc/systemd/system/vault.service /etc/systemd/system/multi-user.target.wants/vault.service ( manual enable vault service)

Note: Make sure you have completed the above process for all other remaining nodes.

Initialize Vault (Only 1st node):

Here we need to initialize the vault. It’s required just once because we have auto-unseal with AWS KMS. If Vault Server is Rebooted, unseal will happen automatically next time.

$ vault operator init

Recovery Key 1: hSZAAt*****************************
Recovery Key 2: AJUK/+**************************
Recovery Key 3: M2nbR****************************
Recovery Key 4: Mac******************************
Recovery Key 5: tC8M*********************************

Initial Root Token: hv*******************************

Success! Vault is initialized

Recovery key initialized with 5 key shares and a key threshold of 3.
 
Please securely store the printed above keys to secret manager.

Check Vault Status (All Nodes):

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.12.1
Build Date               2022-12-27
Storage Type             dynamodb
Cluster Name             vault-cluster
Cluster ID               f3b5f
HA Enabled               true
HA Cluster               https://instance-private-ip:8201
HA Mode                  active
Active Since             2022-12-01

Run below command on other nodes

$ vault status ( other nodes)

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.12.1
Build Date               2022-12-27
Storage Type             dynamodb
Cluster Name             vault-cluster
Cluster ID               599c0
HA Enabled               true
HA Cluster               https://instance-private-ip:8201
HA Mode                  standby
Active Node Address      http://active-instance-ip:8200

Enable Vault Audit :

We can monitor activities and requests to Vault. We can see a timestamp, type of request, who did what, and where for this we have to enable vault audit (recommended for production setup).

$ vault audit enable file -path="vault_audit_prod" file_path=/var/log/vault/audit/vault_audit.log

Success! Enabled the file audit device at: file/

Hurray!! we can see Vault is initialized and unsealed.

Vault Login Demonstration:

We have received the initial root token: hv*****, which can be used for the first login to Vault and to create other users, groups, configurations, etc.
For the test, we can access the Vault UI at http://LB_DNS or https://custome-vault-url.com from our organization network.

As an example, I used web UI to create a new secret.

The process of secret creation is well described in official Vault documentation and no need to write it here.

Conclusion:
There are many different external secret storages that can be used for our applications, one of them is Hashicorp Vault. Vault has enterprise and open-source versions, and can be deployed to different clouds, platforms, and operating systems. In this post, we looked at the deployment Vault in AWS EC2 Servers with DynamoDB as the backend Storage, KMS for auto-unseal, and EBS persistent volumes for audit logs.

In our next blog, we’ll cover how to set up Google Auth as an oidc client for Vault Access.

Thanks for reading this far, and good luck. I appreciate your comments/feedback.

About The Author
Suraj Solanki
DevOps Engineer - II
LinkedIn: https://www.linkedin.com/in/suraj-solanki

Hashicorp Vault Multi Node Cluster Setup on AWS EC2 with DynamoDB

Written by Suraj Solanki