Cloud Driven Development Workshop@devopenspace

This is a live Blog from the workshop “Cloud Driven Development” on https://devopenspace.de/.

Forget a lot of what you know about classic full-stack development. Together, we’ll dive into cloud-driven software development and build a sample serverless application in AWS.

This blog was build live during the workshop on November 2021. So it`s not a complete reference, just a few hints to test and deploy the infrastructure and the applications.

Introduction

In three blocks, we’ll build code and infrastructure with unit and integration testing. Using CDK for AWS, we will build a two-tier application that exemplifies development, automated testing, and deployment. The focus is not on building a great application, but on overcoming the challenges of cloud development. The AWS CDK allows you to define AWS resources and application deployment using C#, F#, Go, Java, JavaScript, Python, TypeScript. I’ll be showing TypeScript and Go. However, you are welcome to use any of the other languages.

In the cloud, in addition to the deployment of the application itself, the automated creation of the infrastructure is added. This presents new challenges for development: How do I do automated testing and debugging?

More Jobs: https://www.tecracer.de/jobs/

More AWS: aws-blog.de

More CDK: https://github.com/tecracer/cdk-templates

More IaC: https://github.com/tecracer/tRick-benchmarks

The Story

The user can copy files to a S3 Bucket. The name of the files will be written in a DynamoDB table with a Lambda function (Block 2). A fargate container reads the table and display the content (Block 3). The Application Load Balancer for the fargate container needs a VPC. This is build in Block 1.

Preparation

You need an AWS User

So first step is the user creation

• Create User

You may start with AdminAccess, but you should narrow the policies down to least privileges.

• Tag, Review skip

• Create User

• Note Access Key, Secret Key

• 

Use the ACCESS_KEY_ID and the Secret to configure the AWS CLI

• Install AWS CLI

Configure AWS CLI Parameters

• Create Access Key
• aws configure --profile admin
• always use the profile parameter with the AWS CLI

Clone

All code is stored in GitHub, so start with cloning.

git clone git@github.com:megaproaktiv/devopenspace.git

Start with the CDK

All steps to start CDK are described in cdkworkshop

Don’t forget to bootstrap yout account (once) with cdk bootstrap.

Make sure to use V2 of the cdk.

Example:

npx cdk@v2.0.0-rc.29 ls --profile admin

Overview

Part 1 - a VPC

Show Unittest and Integration Test with a VPC example.

Part 2 - Serverless

• Unittest, Integration Test, CDK

Part 3 - Autoscaling Fargate Container Service

• Unittest, Local Test

Install-all

https://www.go-on-aws.com/infrastructure-as-go/cdk-go/cdk-go-start/preparation/

• CDK - we use V2

https://docs.aws.amazon.com/cdk/latest/guide/home.html

https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.Vpc.html

Part 1 - VPC

• See also

https://github.com/tecracer/tRick-benchmarks/tree/master/simple-network/cdk

VPC - UnitTest

Assertion CFN (currently CDK V1)

https://docs.aws.amazon.com/cdk/api/latest/docs/assertions-readme.html

https://aws.amazon.com/blogs/developer/testing-cdk-applications-in-any-language/

cd vpc

npm i

npm run test

The test gives errors - try to fix them!

fix test

VPC Integration Test

Do it yourself

go mod init vpc
vi main_test.go 
go mod tidy
go get github.com/aws/aws-sdk-go-v2/aws

Use prepared

go test -v

fix test

Before Deploy

go test
FATA[0000] Template vpc not found
exit status 1
FAIL	vpc	0.627s

Deploy VPC with installed CDK

cdk diff

cdk deploy

vpc: deploying...
[0%] start: Publishing d7bc...fc:current_account-current_region
[100%] success: Published d7...c:current_account-current_region
vpc: creating CloudFormation changeset...

 ✅  vpc

Stack ARN:
arn:aws:cloudformation:eu-central-1:55555555555:stack/vpc/87a240e0-470c-11ec-95cb-02b78989724c

After deploy

cdk ls >stacks.csv

cdkstat

Name                             Status                           Description
----                             ------                           -----------
vpc                              CREATE_COMPLETE                  -

cdkstat vpc

https://github.com/megaproaktiv/cdkstats

Integration Test after deploy

Test, whether the VPC Resource is created.

go test -v

--- FAIL: TestVpcPro (0.59s)
    main_test.go:14:
        	Error Trace:	main_test.go:14
        	Error:      	Not equal:
        	            	expected: "10.0.96.0/21"
        	            	actual  : "10.0.0.0/16"
...
        	Test:       	TestVpcPro
FAIL
exit status 1
FAIL	vpc	0.861s

Try to make test work!

Fix test

Part 2 - Lambda

See for different Lambda examples in several IAC frameworks

https://github.com/tecracer/tRick-benchmarks/tree/master/serverless-standard

Lambda App

The Lambda Application itself.

Test

from base

cd lambda/app

Unittest

go test -v

Fix test

Lambda Infrastructure

cd lambda/infra-ts

npm i

cdk ls

cdk diff

cdkstat

cdk deploy

2 App - Tests

• Level 1 - call Lambda directly from console with json event
• Level 2 - put object in S3 Bucket, which then calls Lambda

* build: 	build go
* fastdeploy: 	Deploy only lambda
* itest: 	call lambda with event
* itest2: 	put item on bucket lambda
* test: 	call go test

Call level 1 test with task itest

Call level 2 test with task itest2

Automated Test process

1. Setup environment: Empty Table
2. Show Table items
3. Call Lambda
4. Show Table items: Check that new items appear

Part 3 - Container

Build a simple container app, which just show the table items.

The Load Balancer routes requests to the running fargate container-

Container test

1. Unit Test the function locally
2. Run Docker locally
3. Run Docker in ECS Cluster

Container APP

Mocks (1)

Test AWS calls with local mocks.

go test -v

vi parameter_test.go

vi query_test.go

App - run

Local Server

go run main/main.go => http://localhost:8080/

Local Container

task docker-run

Copy local Application files

task build deploy

This copies the application files in the infrastructure directory

Container Infrastructure

Deploy the infrastructure

npm i

cdk ls

cdk diff

cdkstat

cdk deploy

Cleanup!

cd container/infra - cdk destroy

cd lambda/infra-ts - cdk destroy

cd vpc - cdk destroy

Cleanup in reverse order because of the dependencies of the stacks.

• Loose Coupling with Parameter Store
• Tight Coupling with Resources

Thanks for participation

Gernot Glawe @megaproaktiv

Learn GO on AWS

• More Jobs: https://www.tecracer.de/jobs/
• More AWS: aws-blog.de
• More CDK: https://github.com/tecracer/cdk-templates
• More IaC: https://github.com/tecracer/tRick-benchmarks

See the full source on github.

Sources

Feedback

For discussion please contact me on twitter @megaproaktiv

Learn more AWS

Want to know more about using AWS? - Meet my colleagues and me in an AWS training