Spice up your Cloudformation Development life

This content is more than 4 years old and the cloud moves fast so some information may be slightly out of date.



Pure CloudFormation coding only with vi can be hard work. But since CloudFormation still is the lingua franca of AWS infrastructure as code, the ecosystem keeps evolving. Even if you only use it indirectly, there are some tools to spice up your development. Hot diagrams, more aroma with compliance checks or bittersweet automated testing. Ready to taste it? Here we go.

Hot diagrams

In many training, I give somebody asks about “can we automatically generate an architecture diagram?” - My usual answer is: “Yes, but it does not make sense!”.

Why is that? Because often auto-generated diagrams do not create clarity but confusion:

Suppose you want to show the differences in routing from the public and the private subnet. You have created a VPC with CloudFormation. If you automatically draw all of the resources, get something like this:

vpc-all

The VPC with all subnets, NACLs and so on…

And this is just the VPC!

This CloudFormation template which is drawn is taken from cloudonaut.

Usually, it makes more sense to create some big picture diagram to discuss the architecture. Just start with a few block which represents resources.

Big picture with excalidraw

A hot tool for this big picture is: excalidraw. It simplifies creation with a small set of keyboards shortcuts. like “r” for rectangle, “t” for text, and “a” for arrows

So in 5 minutes, you got:

big picture

With this diagram, you can more clearly see what you want to discuss. So filtering information and focus is the key.

But Werner said: automate everything!

Filtered generated diagrams with cfn-diagram

Ok, with this really nice tool: cfn-diagramm you have the ability to filter certain resource types.

So take the big VPC diagram, filter some stuff out and you get:

Subnet Routing

This is much clearer and it is generated with cfn-diagram

The Aroma of testing

Do you know taskcat? Its the tool that tests CloudFormation templates. And does linting. And updates AMI maps.

taskcat.

See the code on github. The new version has some neat features, but… you have to configure some things which are not so obvious.

These files have to be configured:

File Content directory
.taskcat.yml  User profile home
.taskcat.yml Test configuration  project
.cfnlintrc.yml  Linting configuration  project

First create your user profile

To get taskcat going, you have to set up a file

~/.taskcat.yml

with e.g.

general:
  auth:
     default: myprofile
  s3_bucket: cdktoolkit-stagingbucket-lobbylibby

Where “myprofile” is a profile from ~/.aws/credentials like:

[myprofile]
aws_secret_access_key=BDVoRtnceo9pzyGmYeIiPaPmofXaykaeB/VOBk3
aws_access_key_id=AKIAIYTTYLLPQSPSPQR

And s3_bucketis the place where taskcat stores all the templates.

Get rid of next error message “mandatory checks”

I am sure there a config for that, but i got rid of the error message in deleting Line 40 from _cfn_lint.py, which is:

   self._rules = cfnlint.core.get_rules(
            self._cfnlint_config.append_rules,
            self._cfnlint_config.ignore_checks,
            self._cfnlint_config.include_checks,
            self._cfnlint_config.configure_rules,
            self._cfnlint_config.include_experimental,
            self._cfnlint_config.mandatory_checks,
        )

It depends on your python installation, should be around here:

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/taskcat/_cfn_lint.py

Configure linter

Now you need a cfnlint config file: .cfnlintrc.yml in the project directory with some code like:

Metadata:
  cfn-lint:
    config:
      regions:
        - eu-central-1
        - eu-west-1
      ignore_checks:
        - E2530
      mandatory-checks:
        - E1002

Configure test to run.

The test are configured in .taskcat.yml.

This is a part of our testing for the tecRacer CloudFormation Training:

project:
  name: cloudformation-training
  owner: gglawe@tecracer.de
  regions:
    - "eu-central-1"
  parameters:
    VPC: "vpc-0f66cf8e7918f109c"
    Subnet: "subnet-0ae1cfddbc50d6506"
    KeyName: "default-lab-key"
    AMI: "ami-0a02ee601d742e89f"
    InstanceType: t3.micro

tests:
  lab00:
    template: lab00-create-stack/lab00-create-stack.template
    regions:
      - "eu-central-1"
  lab01:
    template: lab01-simple-template/lab01-sample-solution.template
    regions:
      - "eu-central-1"

Run taskcat

You run the test with:

taskcat test run 

In the main directory of the project.

taskcat test run -d
 _            _             _
| |_ __ _ ___| | _____ __ _| |_
| __/ _` / __| |/ / __/ _` | __|
| || (_| \__ \   < (_| (_| | |_
 \__\__,_|___/_|\_\___\__,_|\__|



version 0.9.18
[INFO   ] : Lint passed for test default on template /Users/silberkopf/Documents/training/TRC_CloudFormation/aws-training-trc-cloudformation_v2/lab00-create-stack/demo-create-stack.template
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/LICENSE
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/Taskfile.yml
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/lab08-cfn-init/lab08-sample-solution.template
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/lab08-cfn-init/lab08-stub.template
[INFO   ] : ┏ stack Ⓜ tCaT-cloudformation-training-default-db09d27240564b62a212c813f4bc6114                                                                         [INFO   ] : ┣ region: eu-central-1                                                                                                                                  [INFO   ] : ┗ status: CREATE_COMPLETE
[INFO   ] : Reporting on arn:aws:cloudformation:eu-central-1:123456789012:stack/tCaT-cloudformation-training-default-db09d27240564b62a212c813f4bc6114/df86e1e0-afbd-11ea-9490-02946ff1a870
[INFO   ] : Deleting stack: arn:aws:cloudformation:eu-central-1:123456789012:stack/tCaT-cloudformation-training-default-db09d27240564b62a212c813f4bc6114/df86e1e0-afbd-11ea-9490-02946ff1a870
[INFO   ] : Will not delete bucket created outside of taskcat cdktoolkit-stagingbucket-lobbylibby

Taskcat runs all you templates in so many regions as you have configured. If the template was not created and deleted without errors, you will get:

error.

The taskcat_outputs/index.html show the overview.

All outputs are written to taskcat_outputs. All “View Logs” are linked to the local log files of the creating/deletion process.

Auto Update AMI maps

Another nice feature is the update of the AMI maps in all templates which are configured in .taskcat.yml.

The name of the mapping has to be AWSAMIRegionMap. The names are hardwired into the python code, e.g. “AMZNLINUX”.

So if you run the

 taskcat update-ami

In the project main directory, it will change this template code:

	"Mappings": {
		"AWSAMIRegionMap": {
			"eu-west-1": {"AMZNLINUX": "ami-foobar"},
			"eu-central-1": {"AMZNLINUX": "ami-0a9e2b8a093c02922"}
		}
	},

To the updated ami.

	"Mappings": {
		"AWSAMIRegionMap": {
			"eu-west-1": {"AMZNLINUX": "ami-0b4b2d87bdd32212a"},
			"eu-central-1": {"AMZNLINUX": "ami-0a9e2b8a093c02922"}
		}
	},

Check compliance smells with cfn-guard

The newest tool is “cfn-guard”. It is written in rust, you have to install it from Github.

With defined rules the tool checks whether you templates obey these rules.

Example:

AWS::AutoScaling::LaunchConfiguration InstanceType == t3.micro

This rule ensures that only t3.micro is valid for this setting.

At the moment you have to compile the tool for yourself.

With installed rust you may:

git clone git@github.com:aws-cloudformation/cloudformation-guard.git
cd cloudformation-guard
make cfn-guard

And maybe you get an error with

error: failed to parse lock file at ... cfn-guard/Cargo.lock

Just delete the lock file! This is because of different cargo (rust tool) versions.

Create rules

To start with a ruleset a generator is part of the package.json

With make cfn-guard-rulegen (oh another lock…) you generate rules from a template.

We start with a template from /Examples:

  1. cd cloudformation-guard/Examples : Change working directory
  2. ./bin/cfn-guard-rulegen aws-waf-security-automations.template >max.ruleset : Create maximal ruleset
  3. ../bin/cfn-guard -t aws-waf-security-automations.template -r max.ruleset : Test with the ruleset

No error - because we tested against generated ruleset.

Near the end of the max.ruleset we have:

AWS::Lambda::Function Timeout == 300

Now change this line to

AWS::Lambda::Function Timeout == 60

Again:

../bin/cfn-guard -t aws-waf-security-automations.template -r max.ruleset : Test with the ruleset

Now we get:

"[LambdaLogParserFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFBadBotParserFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFCustomResourceFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFHelperFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFReputationListsParserFunction] failed because [Timeout] is [300] and the permitted value is [60]"
Number of failures: 5

This is because we changed the general compliance rule for Lambda Function timeouts to 60 seconds. In the example template we have 5 lambda functions, so we get 5 failures.

You may also check against a set of values:

AWS::Lambda::Function Timeout IN [60, 90, 120]

Summary

So at the start of the development you can use very simple (excalidraw) or generated (cfn-diagram) architecture drawings.

If you have compliance guidelines you may check them before deploying any template with cfn-guard.

Then you can automate your “does it work” testing with taskcat. And you may get rid of updates in AMI maps. Yes, i know of SSM AMI query, but sometimes you need plain cloudformation.

Thanks for reading, please comment with twitter. And also visit our twitch channel: twitch.

Stay healthy in the cloud and on earth!

Thanks

Photo by Payoon Gerinto on Unsplash

Similar Posts You Might Enjoy

Bridging the terraform - CloudFormation gap

CloudFormation does not cover all AWS Resource types. Terraform does a better job in covering resource types just in time. So if you want to use a resource type which CloudFormation does not support yet, but you want to use CloudFormation, you have to build a Custom Resource with an own Lambda Function. CDK to the rescue: use AwsCustomResource. - by Gernot Glawe

Serverless Swagger UI for AWS API Gateway

When implementing REST APIs in AWS there is one service that always comes to mind - Amazon API Gateway. Even though feature-rich, properly documenting your API may become a time-consuming task relatively quickly. In this post, I would like to show you how you can use Swagger UI in combination with a serverless deployment to automatically generate interactive and up-to-date documentation for your APIs. - by Hendrik Hagen

Hostname Resolution and DNS with SAP on AWS

SAP systems running in a distributed environment have certain requirements regarding how to set the hostname and how those need to be resolved from other hosts. In our test landscape we use virtual hostnames to decouple the SAP instances from the underlying hardware which is running on a Red Hat Linux Server. This blog post will walk you through the components in AWS that fullfil those requirements and allow SAP instances to communicate while keeping administrative effort super low. - by Fabian Brakowski