Streamlining Vault Resources Deployment with Terraform and Jenkins CI/CD Pipeline

Vault is a popular open-source secrets management tool that allows you to securely store and access sensitive information such as passwords, tokens, and certificates. In this article, we’ll walk through how to deploy Vault resources using Terraform and automate the process using Jenkins CI/CD.

Prerequisites

Before we begin, you’ll need to have the following prerequisites:

• An AWS account
• The AWS CLI installed on your local machine
• Jenkins installed on a server
• The Terraform CLI is installed on your local machine as well as Jenkins.

Note: This blog post assumes that you already have a Vault cluster set up and running and that you have a basic understanding of how to use Terraform and Jenkins. If you need help setting up a Vault cluster or installing Jenkins, you can refer to our previous blog post on Vault setup or visit the official documentation for Jenkins installation instructions.

Step 1: Create a Terraform project

let’s create a new Terraform project. Create a new directory for your project and create a new file called main.tf. This is where we'll define our Terraform resources.

mkdir vault-terraform
cd vault-terraform
touch main.tf provider.tf variable.tf 

Step 2: Configure the AWS & Vault provider

To use the Vault provider with Terraform, we need to configure the provider in our provider.tf file. Here's an example configuration:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
    vault = {
      source  = "hashicorp/vault"
      version = "~> 3.0"
    }
  }
}

provider "vault" {
  address = "https://vault.example.com" 
  auth_login {
    path = "auth/userpass/login/${var.vault_username}"
    parameters = {
      password = var.vault_password 
    }
  }
}

terraform {
  backend "s3" {
    bucket         = "terraform-state-file-bucket_name"
    key            = "key/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock-table_name"
    encrypt        = true
  }
}

provider "aws" {
  region = "us-east-1"
}

In this example, we're configuring the AWS & Vault provider to connect to a Vault server https://vault.example.com and authenticate using a Vault Userpass method. Replace YOUR_VAULT_USERNAME & YOUR_VAULT_PASSWORDwith your own Vault Creds.

Step 3: Define Vault resources

Now that we've configured the Vault provider, we can define our Vault resources in main.tf. Here’s an example configuration:

Secret

# Create an Secret

resource "vault_generic_secret" "app_secret" {
  path = "secret/my-app-secret"
  data_json = <<EOT
{
    username = "admin"
    password = "secret123"
}
EOT
}

In this example, we're creating a new secret on the path secret/my-app-secret. The secret contains a username and password, which we've defined using the data_json parameter.

Policy

# Create an Policy

resource "vault_policy" "app_policy" {
  name = "my-app-policy"

  policy = <<EOT
path "${var.app_secret_policy_path}"{
  capabilities = ["read"]
}
EOT
}

In this example, we’re creating a policy called my-app-policy. The policy grants read access to the secret at the path secret/my-app-secret.

Authentication method (approle)

# Create an AppRole

resource "vault_approle_auth_backend_role" "app_approle" {
  backend        = "approle"
  role_name      = "my-app-role"
  bind_secret_id = true
  token_policies = [vault_policy.app_policy.name]
}

# Create an Secret_id

resource "vault_approle_auth_backend_role_secret_id" "app_secret_id" {
  backend   = vault_approle_auth_backend_role.app_approle.backend
  role_name = vault_approle_auth_backend_role.app_approle.role_name
}

# Create an Client Token

resource "vault_approle_auth_backend_login" "login" {
  backend   = vault_approle_auth_backend_role.app_approle.backend
  role_id   = vault_approle_auth_backend_role.app_approle.role_id
  secret_id = vault_approle_auth_backend_role_secret_id.app_secret_id.secret_id
}

In this example, we’re creating a new approle backend at the path approle and also configuring the policy to authenticate against an approle token.

Next, Then we create an AppRole client token in Vault using the vault_approle_auth_backend_role resource. We're also generating a secret ID for the role using the vault_approle_auth_backend_role_secret_id resource.

Secret Manager

# Create an secret manager
resource "aws_secretsmanager_secret" "vault_approle_token" {
  name = "vault_approle_token"
}

resource "aws_secretsmanager_secret_version" "vault_approle_token" {
  secret_id = aws_secretsmanager_secret.vault_approle_token.id
  secret_string = vault_approle_auth_backend_login.login.client_token
}

We then store the secret ID in the AWS Secrets Manager using the aws_secretsmanager_secret aws_secretsmanager_secret_version resources. The aws_secretsmanager_secret the resource creates a new secret with the name "vault_approle_token", and the aws_secretsmanager_secret_version resource creates a new version of the secret with the actual secret ID value.

You can utilize these client tokens from the secret manager to retrieve application secrets from the vault and incorporate it into your external application.

Note: You can improve the reusability and flexibility of your Terraform code by adding a variables.tf file. This file allows you to parameterize your Terraform templates, making it easy to customize them for different environments or use cases. By defining variables in a separate file, you can avoid hardcoding values in your code and keep them in a centralized location, making it easier to manage and maintain them over time.

Step 4: Set up a Jenkins job

Here are the steps to set up a Jenkins job:

1. Open the Jenkins web interface and click on “New Item” to create a new job.
2. Enter a name for the job, choose “Pipeline” as the job type, and click “OK”.
3. Scroll down to the “Pipeline” section and select “Pipeline script from SCM” as the definition.
4. Choose “Git” as the SCM and enter the URL for your Git repository where the Jenkinsfile and Terraform code is located.
5. In the “Script Path” field, enter the path to your Jenkinsfile. For example, if your Jenkinsfile is located in the root of the repository, you can just enter “Jenkinsfile”.
6. Save the job configuration and run the job to verify that it works.

Here’s an example of a Sample Jenkinsfile:

pipeline {
    // agent any

    environment {
        // add environment if you are going to store vault-creds on jenkins credential
    }

     parameters {
            // add if you are making job paramterized
    }

    stages {

        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Terraform Init') {
            steps {
                sh 'terraform init'
            }
        }

        stage('Terraform Plan') {

            steps {
                sh 'terraform plan -out=tfplan'
            }
        }

        stage('Terraform Apply') {

            steps {
                input 'Do you want to proceed with the deployment?'
                sh 'terraform apply tfplan -auto-approve'

                script {
                    if (currentBuild.result == 'FAILED') {
                        sh 'terraform destroy -auto-approve'
                        
                    }
                }
            }
        }
    }
}

Conclusion:
Deploying Vault resources with Terraform and Jenkins provides an automated and repeatable approach to managing secrets and secure infrastructure. With Terraform, we can define our Vault resources as code, version control them, and manage their lifecycle using a continuous integration and delivery (CI/CD) pipeline powered by Jenkins.

Thank you for reading this far! I hope you found this blog post informative and useful for your Vault resources deployment. I welcome any comments or feedback.

About The Author
Suraj Solanki
DevOps Engineer — II
LinkedIn: linkedin.com/in/suraj-solanki
Book your time with me at: topmate.io/suraj_solanki