Getting around circular CloudFormation Dependencies: S3-Event-Lambda-Role

This content is more than 4 years old and the cloud moves fast so some information may be slightly out of date.



Getting around circular CloudFormation dependencies

Several posts complain about the inability of CloudFormation to apply a Lambda event function to an S3 Bucket with an dynamically generated name.

The standard UseCase is an S3 Bucket with a Lambda event notification. In this special case the Bucket has a dynamically generated name. This cannot be done by pure CloudFormation!

How to work around this circular depency? Let me show you an easy way:

In AWS Blog: Handling circular dependency errors in AWS CloudFormation [AWS1] and serverless hero Ben Kehoe writes here .[BEN1]

This will not work

So in pure CloudFormation this is not possible:

BucketEvent

That is because we have a circular dependency. As [AWS1] says: “The bucket notification is dependent on the Lambda function and the Lambda function is dependent on the execution role, which is dependent on the S3 bucket.”

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  Bucket:
    Type: AWS::S3::Bucket     
  Function:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: s3://bucketname/object.zip # Add in an S3 URI where you have code Lambda Code
      Runtime: python2.7
      Handler: index.handler
      Policies:
        - Version: 2012-10-17
          Statement: 
            - Effect: Allow
              Action: s3:GetObject*
              Resource: !Sub "arn:aws:s3:::${Bucket}*"
      Events:
        Upload:
          Properties:
            Bucket:
              Ref: Bucket
            Events: s3:ObjectCreated:*
          Type: S3

So we have to decouple things. We can do that by first creating the bucket and the lambda function and bringing them together after that.

The way to extend CloudFormation is a custom resource. That is a Lambda Function wich acts as a CloudFormation resource. This “bring together” function gets notified to create or update or delete itself.

You could also do this as an AWS cli script after creation of the CloudFormation stack. But if you do that, you won’t have the stack itself as a whole functional unit. But beeing lazy, the question is: do I really have to code this all by myself? - Answer is no - CDK to the rescue.

This will work

BucketEvent

CDK code

If you create an CDK app with cdk init app --language=typescript you just have to get some imports:

import {Bucket,EventType} from '@aws-cdk/aws-s3';
import {LambdaDestination} from '@aws-cdk/aws-s3-notifications';
import {Function, Runtime, Code} from '@aws-cdk/aws-lambda'

And replace // The code that defines your stack goes here with this snippet:


export class CfnGraphStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);


    // S3
    const bucket = new Bucket(this, "testbucketmpa",{
      removalPolicy: cdk.RemovalPolicy.DESTROY, // NOT recommended for production code
    });

     // Lambda
    const lambda = new Function(this, 'HelloHandler', {
      code: Code.asset(path.join(__dirname,  '../lambda')),
      handler: 'hello.handler',
      runtime: Runtime.NODEJS_8_10,
      memorySize: 1024
    });

    // S3 -> Lambda
    bucket.addEventNotification(EventType.OBJECT_CREATED, new LambdaDestination(lambda));


  }
}

Of course there have to be a node Lambda function in the dir lambda

The CDK has already added the “bring together” Lambda function. And it is all happening under the hood, we just add the EventNotifcation in the ts code to the Bucket.

Generated template

 testbucketmpaNotificationsB2722499:
    Type: Custom::S3BucketNotifications
    Properties:
  ...
      BucketName:
        Ref: testbucketmpaAE8E5392
      NotificationConfiguration:
        LambdaFunctionConfigurations:
          - Events:
              - s3:ObjectCreated:*
            LambdaFunctionArn:
              Fn::GetAtt:
                - HelloHandler2E4FBA4D
                - Arn
    DependsOn:
      - HelloHandlerAllowBucketNotificationsFromCfnGraphStacktestbucketmpa0C1D72A46851358B

This snippet from the generated CloudFormation (cdk synth) shows the Custom Ressource which waits (with DependsOn until the AWS::Lambda::Permission is created.)

The Lambda Function is created inline:

BucketNotificationsHandler050a0587b7544547bf325f094a3db8347ECC3691:
    Type: AWS::Lambda::Function
    Properties:
      Description: AWS CloudFormation handler for "Custom::S3BucketNotifications" resources (@aws-cdk/aws-s3)
      Code:
        ZipFile: >-
          exports.handler = (event, context) => {
              const s3 = new (require('aws-sdk').S3)();
              const https = require("https");
              const url = require("url");
              log(JSON.stringify(event, undefined, 2));          
...
              return s3.putBucketNotificationConfiguration(req, (err, data) => {

Its creates the BucketNotificatioConfiguration and you’re done!

Similar Posts You Might Enjoy

More Tools - CDK Examples

We need more CDK examples In this github repo we focus on examples for every day work. While there are some nice examples for the fancy stuff like fargate, ecs and so on in aws-cdk-examples/typescript at master · aws-samples/aws-cdk-examples · GitHub, i felt that basic examples where missing. So we created GitHub - tecracer/cdk-templates: Templates for aws cdk - by Gernot Glawe

tRick: simple network 1 - Abstraktion und LoC

Vergleich Infrastructure as Code (IaC) Frameworks - tRick Ein Toolvergleich für Infrastructure as Code. Um effektiv AWS oder generell Cloud Ressourcen zu erzeugen, verwendet man zur Erhöhung des Automatisierungsgrades “Infrastracture as Code”, d.h. die Server, Datenbanken usw. werden in einer Sprache kodiert. Dieser Vorgang wird sinvollerweise über ein Framework, welches Tools dafür zur Verfügung stellt unterstützt. Aber für welches Tool/Framework entscheidet man sich? Hier werden wir mit Dimensionen für den tRick Benchmark Entscheidungshilfen geben. - by Gernot Glawe

Große Wolken - CloudFormation Makros

How to: CloudFormation Makro CloudFormation vermisst gegenüber Terraform einige Funktionen, die das Erstellen von Infrastruktur vereinfachen können. Das ist grundsätzlich korrekt, allerdings gibt es in CloudFormation die Möglichkeit, sich selber um den Einbau solcher Funktionen zu kümmern. Das geht mithilfe sogenannter CloudFormation Makros. CloudFormation Makros sind Funktionen, die wir per CloudFormation erstellen können und dann in weiteren CloudFormation Templates einbauen und verwenden können. Wir zeigen dies am Beispiel einer Count Funktion. - by David Schellenburg , André Reinecke