Spice up your Cloudformation Development life

Pure CloudFormation coding only with vi can be hard work. But since CloudFormation still is the lingua franca of AWS infrastructure as code, the ecosystem keeps evolving. Even if you only use it indirectly, there are some tools to spice up your development. Hot diagrams, more aroma with compliance checks or bittersweet automated testing. Ready to taste it? Here we go.

Hot diagrams

In many training, I give somebody asks about “can we automatically generate an architecture diagram?” - My usual answer is: “Yes, but it does not make sense!”.

Why is that? Because often auto-generated diagrams do not create clarity but confusion:

Suppose you want to show the differences in routing from the public and the private subnet. You have created a VPC with CloudFormation. If you automatically draw all of the resources, get something like this:

The VPC with all subnets, NACLs and so on…

And this is just the VPC!

This CloudFormation template which is drawn is taken from cloudonaut.

Usually, it makes more sense to create some big picture diagram to discuss the architecture. Just start with a few block which represents resources.

Big picture with excalidraw

A hot tool for this big picture is: excalidraw. It simplifies creation with a small set of keyboards shortcuts. like “r” for rectangle, “t” for text, and “a” for arrows

So in 5 minutes, you got:

With this diagram, you can more clearly see what you want to discuss. So filtering information and focus is the key.

But Werner said: automate everything!

Filtered generated diagrams with cfn-diagram

Ok, with this really nice tool: cfn-diagramm you have the ability to filter certain resource types.

So take the big VPC diagram, filter some stuff out and you get:

This is much clearer and it is generated with cfn-diagram

The Aroma of testing

Do you know taskcat? Its the tool that tests CloudFormation templates. And does linting. And updates AMI maps.

.

See the code on github. The new version has some neat features, but… you have to configure some things which are not so obvious.

These files have to be configured:

First create your user profile

To get taskcat going, you have to set up a file

~/.taskcat.yml

with e.g.

general:
  auth:
     default: myprofile
  s3_bucket: cdktoolkit-stagingbucket-lobbylibby

Where “myprofile” is a profile from ~/.aws/credentials like:

[myprofile]
aws_secret_access_key=BDVoRtnceo9pzyGmYeIiPaPmofXaykaeB/VOBk3
aws_access_key_id=AKIAIYTTYLLPQSPSPQR

And s3_bucketis the place where taskcat stores all the templates.

Get rid of next error message “mandatory checks”

I am sure there a config for that, but i got rid of the error message in deleting Line 40 from _cfn_lint.py, which is:

   self._rules = cfnlint.core.get_rules(
            self._cfnlint_config.append_rules,
            self._cfnlint_config.ignore_checks,
            self._cfnlint_config.include_checks,
            self._cfnlint_config.configure_rules,
            self._cfnlint_config.include_experimental,
            self._cfnlint_config.mandatory_checks,
        )

It depends on your python installation, should be around here:

~/.pyenv/versions/3.7.3/lib/python3.7/site-packages/taskcat/_cfn_lint.py

Configure linter

Now you need a cfnlint config file: .cfnlintrc.yml in the project directory with some code like:

Metadata:
  cfn-lint:
    config:
      regions:
        - eu-central-1
        - eu-west-1
      ignore_checks:
        - E2530
      mandatory-checks:
        - E1002

Configure test to run.

The test are configured in .taskcat.yml.

This is a part of our testing for the tecRacer CloudFormation Training:

project:
  name: cloudformation-training
  owner: gglawe@tecracer.de
  regions:
    - "eu-central-1"
  parameters:
    VPC: "vpc-0f66cf8e7918f109c"
    Subnet: "subnet-0ae1cfddbc50d6506"
    KeyName: "default-lab-key"
    AMI: "ami-0a02ee601d742e89f"
    InstanceType: t3.micro

tests:
  lab00:
    template: lab00-create-stack/lab00-create-stack.template
    regions:
      - "eu-central-1"
  lab01:
    template: lab01-simple-template/lab01-sample-solution.template
    regions:
      - "eu-central-1"

Run taskcat

You run the test with:

taskcat test run 

In the main directory of the project.

taskcat test run -d
 _            _             _
| |_ __ _ ___| | _____ __ _| |_
| __/ _` / __| |/ / __/ _` | __|
| || (_| \__ \   < (_| (_| | |_
 \__\__,_|___/_|\_\___\__,_|\__|



version 0.9.18
[INFO   ] : Lint passed for test default on template /Users/silberkopf/Documents/training/TRC_CloudFormation/aws-training-trc-cloudformation_v2/lab00-create-stack/demo-create-stack.template
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/LICENSE
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/Taskfile.yml
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/lab08-cfn-init/lab08-sample-solution.template
[S3: -> ] s3://cdktoolkit-stagingbucket-lobbylibby/cloudformation-training/lab08-cfn-init/lab08-stub.template
[INFO   ] : ┏ stack Ⓜ tCaT-cloudformation-training-default-db09d27240564b62a212c813f4bc6114                                                                         [INFO   ] : ┣ region: eu-central-1                                                                                                                                  [INFO   ] : ┗ status: CREATE_COMPLETE
[INFO   ] : Reporting on arn:aws:cloudformation:eu-central-1:123456789012:stack/tCaT-cloudformation-training-default-db09d27240564b62a212c813f4bc6114/df86e1e0-afbd-11ea-9490-02946ff1a870
[INFO   ] : Deleting stack: arn:aws:cloudformation:eu-central-1:123456789012:stack/tCaT-cloudformation-training-default-db09d27240564b62a212c813f4bc6114/df86e1e0-afbd-11ea-9490-02946ff1a870
[INFO   ] : Will not delete bucket created outside of taskcat cdktoolkit-stagingbucket-lobbylibby

Taskcat runs all you templates in so many regions as you have configured. If the template was not created and deleted without errors, you will get:

.

The taskcat_outputs/index.html show the overview.

All outputs are written to taskcat_outputs. All “View Logs” are linked to the local log files of the creating/deletion process.

Auto Update AMI maps

Another nice feature is the update of the AMI maps in all templates which are configured in .taskcat.yml.

The name of the mapping has to be AWSAMIRegionMap. The names are hardwired into the python code, e.g. “AMZNLINUX”.

So if you run the

 taskcat update-ami

In the project main directory, it will change this template code:

	"Mappings": {
		"AWSAMIRegionMap": {
			"eu-west-1": {"AMZNLINUX": "ami-foobar"},
			"eu-central-1": {"AMZNLINUX": "ami-0a9e2b8a093c02922"}
		}
	},

To the updated ami.

	"Mappings": {
		"AWSAMIRegionMap": {
			"eu-west-1": {"AMZNLINUX": "ami-0b4b2d87bdd32212a"},
			"eu-central-1": {"AMZNLINUX": "ami-0a9e2b8a093c02922"}
		}
	},

Check compliance smells with cfn-guard

The newest tool is “cfn-guard”. It is written in rust, you have to install it from Github.

With defined rules the tool checks whether you templates obey these rules.

Example:

AWS::AutoScaling::LaunchConfiguration InstanceType == t3.micro

This rule ensures that only t3.micro is valid for this setting.

At the moment you have to compile the tool for yourself.

With installed rust you may:

git clone git@github.com:aws-cloudformation/cloudformation-guard.git
cd cloudformation-guard
make cfn-guard

And maybe you get an error with

error: failed to parse lock file at ... cfn-guard/Cargo.lock

Just delete the lock file! This is because of different cargo (rust tool) versions.

Create rules

To start with a ruleset a generator is part of the package.json

With make cfn-guard-rulegen (oh another lock…) you generate rules from a template.

We start with a template from /Examples:

1. cd cloudformation-guard/Examples : Change working directory
2. ./bin/cfn-guard-rulegen aws-waf-security-automations.template >max.ruleset : Create maximal ruleset
3. ../bin/cfn-guard -t aws-waf-security-automations.template -r max.ruleset : Test with the ruleset

No error - because we tested against generated ruleset.

Near the end of the max.ruleset we have:

AWS::Lambda::Function Timeout == 300

Now change this line to

AWS::Lambda::Function Timeout == 60

Again:

../bin/cfn-guard -t aws-waf-security-automations.template -r max.ruleset : Test with the ruleset

Now we get:

"[LambdaLogParserFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFBadBotParserFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFCustomResourceFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFHelperFunction] failed because [Timeout] is [300] and the permitted value is [60]"
"[LambdaWAFReputationListsParserFunction] failed because [Timeout] is [300] and the permitted value is [60]"
Number of failures: 5

This is because we changed the general compliance rule for Lambda Function timeouts to 60 seconds. In the example template we have 5 lambda functions, so we get 5 failures.

You may also check against a set of values:

AWS::Lambda::Function Timeout IN [60, 90, 120]

Summary

So at the start of the development you can use very simple (excalidraw) or generated (cfn-diagram) architecture drawings.

If you have compliance guidelines you may check them before deploying any template with cfn-guard.

Then you can automate your “does it work” testing with taskcat. And you may get rid of updates in AMI maps. Yes, i know of SSM AMI query, but sometimes you need plain cloudformation.

Thanks for reading, please comment with twitter. And also visit our twitch channel: twitch.

Stay healthy in the cloud and on earth!

Thanks

Photo by Payoon Gerinto on Unsplash