Our teams working with Terraform ofttimes find that information technology is helpful to create a variable hierarchy that supports defining global, environment, and stack-specific variables. While this is a fairly bones concept, Terraform not only doesn't requite you a way to exercise it out of the box only actively works against supporting the concept, despite strong user desire.

Chris, BTI360 engineer and author of thirstydeveloper.io shares in this post how teams at BTI360 implement a Terraform variable hierarchy using Terragrunt and YAML files.

Case Infrastructure

Permit's say nosotros have a simple two-stack, two-surround terraform project using Terragrunt. Information technology might look something similar this:

deployments/   dev/                # dev environs stacks     database/         # dev environment database stack       terragrunt.hcl  # dev surroundings database stack terragrunt config     webserver/        # dev environment webserver stack       terragrunt.hcl  # dev environment webserver stack terragrunt config   prod/               # production environment stacks     database/       terragrunt.hcl     webserver/       terragrunt.hcl     root.hcl            # parent terragrunt config file modules/   stacks/             # stacks deployed by deployments     database/       chief.tf         # terraform manifest for database stack     webserver/       main.tf         # terraform manifest for webserver stack  with each of the terragrunt.hcl files including shared terragrunt configuration from the root.hcl with:  # terragrunt.hcl include {   path = find_in_parent_folders("root.hcl") }

Now, say both the database and webserver stacks have in a project_name variable that is always set to sample_project. We could create a terraform.tfvars file in each deployment directory to set this variable value, something like:

deployments/   dev/                     database/                terraform.tfvars # dev database variable values       terragrunt.hcl       webserver/               terraform.tfvars # dev webserver variable values       terragrunt.hcl     prod/                    database/       terraform.tfvars # prod database variable values       terragrunt.hcl     webserver/       terraform.tfvars # prod webserver variable values       terragrunt.hcl     root.hcl

where each terraform.tfvars file contains:

project_name = "sample_project"

Following the DRY master however, nosotros'd prefer to but ascertain this variable once at a global level, perhaps with a terraform.tfvars file underneath deployments/:

deployments/   dev/                   prod/                  terraform.tfvars    # global variable values   root.hcl            # parent terragrunt config file

Then we could load this root terraform.tfvars with the root.hcl similar so:

terraform {   ...   extra_arguments "load_global_variables" {     commands = get_terraform_commands_that_need_vars()     optional_var_files = ["${get_parent_terragrunt_dir()}/terraform.tfvars"]   } }

This, however, would be a mistake.

Why tfvars Don't Work

The tfvars approach higher up worked well until Terraform 0.12, which introduced a controversial feature to print warnings when values are specified for undefined variables. For instance, if we add an unused variable to deployments/terraform.tfvars:

# deployments/terraform.tfvars project_name = "sample_project" unused = true

whatsoever stack we plan or apply now prints:

Alarm: Value for undeclared variable The root module does non declare a variable named "unused" only a value was found in file...

The warnings are annoying enough in that they ataxia the programme output, just worse nonetheless the warning states:

Using a variables file to set up an undeclared variable is deprecated  and will go an error in a future release.

That means we can't reliably utilize tfvars files for a variable hierarchy, at to the lowest degree not if yous have any variables divers that go unused past whatever stacks.

This poses a trouble considering it doesn't take long for an infrastructure project to end upwards with some variables defined at some parent level that some stacks beneath don't demand or use, as many issue comments point out.

Hashicorp does give you a workaround. You lot tin can specify variable values using environment variables, TF_VAR_project_name=sample_project for example, simply Terraform leaves the challenge of easily and reproducibly defining those environment variables across systems every bit an exercise to the developer.

We can practise better.

Variable Hierarchy with Terragrunt Inputs

Terragrunt offers two key capabilities that offer usa a fashion out.

The first is Terragrunt's inputs attribute, which accepts an HCL object and translates each fundamental/value pair into an surroundings variable passed to Terraform.

The 2nd is that Terragrunt allows access to all of Terraform's built-in functions within the terragrunt.hcl / root.hcl files. Specifically, nosotros have access to:

  • file – for loading file contents to a string
  • yamldecode – for converting a YAML cord to an HCL object
  • fileexists – for checking if a file exists earlier attempting to load it
  • merge – for merging multiple HCL objects

By combining these functions with Terragrunt's inputs attribute, we tin can establish our variable bureaucracy using YAML files. Here'south how:

Start by converting the terraform.tfvars files to config.yml files1. For case, convert:

# deployments/terraform.tfvars project_name = "sample_project" unused = true

to:

# deployments/config.yml  ---  project_name: "sample_project"  unused: true

You lot'll cease upwardly with a structure looking like:

dev/   database/     config.yml      # dev/database variable values     terragrunt.hcl   webserver/     config.yml      # dev/webserver variable values     terragrunt.hcl   config.yml        # dev environment variable values prod/   database/     config.yml      # prod/database variable values     terragrunt.hcl   webserver/     config.yml      # prod/webserver variable values     terragrunt.hcl   config.yml        # prod environs variable values config.yml          # global variable values root.hcl

Next, modify your root.hcl to:

  1. Recursively observe every config.yml between the root.hcl and the terragrunt.hcl
  2. Load each config.yml file that exists into a cord using file()
  3. Catechumen each YAML string into a HCL object using yamldecode()
  4. Merge all the objects using merge(), ensuring lower-level objects override higher-level ones
  5. Pass the merged object into inputs

Here'due south a sample root.hcl that does the above:

# root.hcl locals {   root_deployments_dir       = get_parent_terragrunt_dir()   relative_deployment_path   = path_relative_to_include()   deployment_path_components = compact(carve up("/", local.relative_deployment_path))    # Get a list of every path between root_deployments_directory and the path of   # the deployment   possible_config_dirs = [     for i in range(0, length(local.deployment_path_components) + i) :     join("/", concat(       [local.root_deployments_dir],       piece(local.deployment_path_components, 0, i)     ))   ]    # Generate a list of possible config files at every possible_config_dir   # (support both .yml and .yaml)   possible_config_paths = flatten([     for dir in local.possible_config_dirs : [       "${dir}/config.yml",       "${dir}/config.yaml"     ]   ])    # Load every YAML config file that exists into an HCL object   file_configs = [     for path in local.possible_config_paths :     yamldecode(file(path)) if fileexists(path)   ]    # Merge the objects together, with deeper configs overriding higher configs   merged_config = merge(local.file_configs...) }  # Pass the merged config to terraform every bit variable values using TF_VAR_ # environs variables inputs = local.merged_config

With this approach, we tin:

  1. Put global variables in deployments/config.yml
  2. Put environment-specific variables in deployments/dev/config.yml and deployments/prod/config.yml
  3. Put stack specific variables in the config.yml next to the corresponding terragrunt.hcl

Furthermore, nosotros can override values in higher-level config files by redefining them at lower-levels. For instance, we could redefine project_name in deployments/dev/webserver/config.yml every bit:

# deployments/dev/webserver/config.yml project_name: webserver

and that would override the value of sample_project divers in deployments/config.yml.

Using YAML files as our Terraform variable hierarchy has proved very successful for our teams. We hope it helps yours besides. For more information on this approach, including a fully worked example, meet Chris' terraform-skeleton series on thirstydeveloper.io. Good luck!

Footnotes

  1. The YAML loading doesn't play nice with config.yml files that are empty or contain merely a --- start of document marking. Y'all tin can delete the empty config.yml file and everything will piece of work, due to the fileexists check. If you prefer to keep the empty config.yml, the most minimal contents required are:
--- {}

Interested in Solving Challenging Issues? Work Hither!

Are yous a software engineer, interested in joining a software company that invests in its teammates and promotes a potent engineering culture? So you're in the right identify! Check out our current Career Opportunities. We're always looking for like-minded engineers to join the BTI360 family.

Related Posts:

  • Organizing Terraform Code with Terragrunt
  • Advantages and Limitations of Terragrunt-Managed Backends
  • Managing Terraform Remote State with CloudFormation