Testing Terraform with InSpec (Part 1)

While Infrastructure-as-Code slowly becomes omnipresent, many of the communicated advantages of the approach stay mostly unrealized. Sure, code style checks (linting) and even automated documentation get more common every month. But one of the cornerstones often gets ignore: testing.

Let’s see which types of code testing are available and how to do it without writing too much code.

The promise of the Infrastructure-as-Code (short: IaC) movement is to handle infrastructure just as if it was a program. And adapt all the decades-old techniques for software engineering to get the most out of it.

But as with any new technology, it is hard to realize all of the promised benefits. Many are well understood, others widely used but some provide elusive.

Realized Benefits

To put one thing clear: Using IaC has massive advantages to manual installations and many of them are easy to realize - as long as you know basics of GIT (or another version control system you might prefer).

Putting all your code into something like GIT immediately gives you features like traceability (who added what code at what point in time), rollback capability (undo code changes) and versioning of code (whether you use GIT tags or not). All this helps making your infrastructure setup more transparent.

Even if you, at some point in time, want to change technologies or platforms you have an advantage here: you know that the code in your version control is exactly the one which created your systems. And you could still transpale this into another system, more or less. If you achieved the next steps: Reproducability and parametrization.

These are actually some of the first steps where you can fail easily. Are there any manual steps involved with your setup? Do you rely on a specific AMI to create your systems (AMIs might get removed for various reasons)? Did you hardcode AWS regions, IP ranges, …?

There is no general rule to address these items, many are connected to your actual use case. In this area, only practice makes perfect. Even if you find a magical 10-step plan on making your deployments reproducible, you will likely only live by the rules if you repeatedly failed following them.

Advanced Benefits

As soon as you work on code in a team, you will notice it will start to look non-uniform. If everybody in the team uses a different type of code indentation, different naming of components or confusing constructs (ternary operator, anyone?) things will escalate quickly. People do not understand the code anymore, hesitate to modify it or even include case distinctions to switch between old code and their own one.

This is the domain of Linters and Fixers. These are tools which programmers love (read: disdain) which either prohibit using “unclean” constructs and formats in your code or adjust it to some community style guidelines. Why disdain? Well, do you want a colleague sitting next to you and comment about every typo in your code, even if he is right? But still, these tools are for the greater good - keeping your code clean and future-proof.

Another quality gate are code reviews. People do not commit directly to version control, but they establish a process where colleagues have a look onto the changes and comment on quality, possible errors and alternative approaches. Only if you reach consensus on the code being appropriate, well-formed and suitable for long-term use, it will get merged into your common code base (often called a Pull Request or Merge Request).

But if your code/infrastructure is complex, you need testing. And in many cases, manual testing will not cut it. If you have to check every one of the 20 different ways your parametrized code could execute, you will inavertedly miss one of them or even only test your new code.

Types of Testing

Software testing is a whole profession by itself and has produced great books like “Agile Testing: A Practical Guide for Testers and Agile Teams” or “More Agile Testing: Learning Journeys for the Whole Team” by Janet Gregory and Lisa Crispin. While it’s worth reading and provides many interesting insights, it might be slightly out of scope for a blog post.

To put things short, there are different test types (often sorted into testing quadrants) which address different topics. Handling of edge conditions, correct execution, matching of expectations and functionality, behaviour under scalability or security aspects, etc.

For Infrastructure-as-Code I would simplify it as follows:

Unit Testing

Testing code without actually integrating with anything - this is the domain of IaC concepts which rely on programming languages. For example, you can use the unit testing frameworks which come along with Typescript, Python or Go if you are using CDK.

These unit tests are supposed to run quickly to test hundreds of execution paths within a few seconds. This can only be achieved by not calling out to AWS APIs, but instead sending back canned responses from the unit testing framework.

Integration Testing

In this step, actual infrastructure gets created. As these steps are slower, they will just be executed for some of the main use cases to test code changes. Some companies test all paths in periodic “nightly”/“weekly” tests and respond if one of the cases is broken.

If your code creates a VPC, subnets, an IGW, NAT Gateways and some S3 Endpoint, your test will execute it and then check if your assumptions are met. Maybe your forgot some subnet? That test will fail and you know that you have to fix this.

A problem with this is, that you write the same thing twice. You write code for creating a VPC and code for testing if it exists. While this seems like a waste of time, it actually avoids mistakes by typos or wrong assumptions. If you want to know how to write proper tests, check out “The Art of Unit Testing” by Ray Osherove - which is adaptable to integration tests as well.

LocalStack

LocalStack is a mocking framework to imitate AWS APIs for speedy, local testing of solutions. In my opinion, this solutions falls between Unit and Integration testing, as it is a big component you need to install and integrate it.

Personally, I would rather execute unit tests against my own mocks and later test AWS in an integration stage. This can be cleanly isolated using an old, but repopularized pattern called Hexagonal Architecture. It recently saw a rise in popularity with blog entries from The Netflix Tech Blog and the AWS Compute Blog.

But in the end, it’s all about the specific use cases and your preferences.

End-to-End Testing

Why test more, if we know that integration was successful? Because we tested the building block’s existence, not their functionality.

In our VPC example above, did you notice I did not mention route tables? So unless you try to access an instance within the VPC you will not notice something is missing. Your oversight is due to not testing the actual functionality.

What if you could create an EC2 instance, automatically log into it and check if outgoing web requests work? That would greatly increase your code quality as you can cover parts of the actual user journey who would use instances in that VPC in production.

Integration Tests with InSpec / Test Kitchen

In unexpected curveball, one of the solutions can be found in the area of Configuration Management. While this area also is part of Infrastructure-as-Code, it is more involved in setting up Operating Systems (OS) and applications on instances. But it yielded two tools which help a lot with Integration/Functional Testing.

InSpec

InSpec is a domain specific language (DSL) which is connected with Chef (recently purchased by Progress Software). While Chef Infra is the software for all those OS-level tasks, InSpec is its counterpart for testing:

describe port(80) do
  it { should be_listening }
end

describe http('http://localhost/') do
  its('status') { should eq 200 }
  its('body') { should include 'Welcome' }
end

You can see how this bundles both testing types into one test profile. Is the port open? And does it return the right content?

This is not limited to EC2 instance testing, as InSpec has resources for AWS as well:

describe aws_vpc('vpc-123456789') do
  it { should be_available }

  its('cidr_block') { should cmp '10.0.0.0/16' }
  its('instance_tenancy') { should eq 'default' }
end

There are even ready-made InSpec tests to check if an AWS Account is matching best practices.

Test Kitchen

While InSpec sounds nice, for testing you do not want to write huge shell scripts applying your Terraform/CDK infrastructure, executing code and then destroying it.

This is where Test Kitchen comes into play, another community tool by Chef/Progress Software. Just as with InSpec it was first designed to manage VMs/EC2 Instances but got extended later on, specifically with the Terraform Provisioner.

With a simple configuration file (kitchen.yml) in your project you can wire everything up so that Test Kitchen creates your Terraform infrastructure, executes various InSpec tests against it, tears it down and reports all successes/failures to you.

You can even include a step of logging into created EC2 instances and do functional checks on them.

Part 2: Practical Walkthrough

This post was very heavy on the theory side, but it is important to understand the challenges and basic terms to value Integration testing of IaC repositories. Too often people say that infrastructure does not need much testing or manual testing is enough - or that testing it automatically is too hard.

I agree on the last part, if there is a clear distrintion between tools: If you have operations people working with descriptive languages like Terraform/Ansible and then require them to write Go-based code for Terratest - that is a tough sell.

On the other hand, if your infrastructure is managed by software engineers with CDK then asking them to write tests in their preferred language is not too big of a deal.

If you want to know more about CDK and testing that one, look at the blog acticle by my colleague Gernot.

For the complete walkthrough of Terraform, VSCode Dev Containers, Test Kitchen and InSpec check out the second part of this post.