The case of the missing bucket notifications

This content is more than 4 years old and the cloud moves fast so some information may be slightly out of date.

The case of the missing bucket notifications

A few days ago I was trying to do something quite simple. I wanted to send S3 Put-Events to multiple Lambda functions for processing. This is a pretty common pattern. To implement it you have to use an architecture such as the one you can see in the title image above.

This is because S3 has a limit on the event handlers (notification targets) per event type of exactly one. That’s not a lot. To work around that it’s common to use SNS as an intermediary to fan out the event notifications to multiple targets.

I had done the same as part of a larger infrastructure but for some reason my lambdas weren’t getting any events from S3.

I tried a lot to figure out the issue, but everything seemed to be configured correctly.

  • The deployment pipeline was deploying my CloudFormation template correctly
  • The notification was configured correctly in the S3 GUI.
  • Even after changing the name of the notification in the GUI the S3-check still reported it as being configured correctly
  • Publishing other events to SNS triggered my Lambdas

So it seemed like the problem was the communication between S3 and SNS. That was bad, because there’s not a lot of logs (none) you can look at.

I built the whole thing manually in my personal account via the GUI and everything seemed fine - all notifications were being triggered as expected. That annoyed me a little to say the least so I finally decided to extract the involved parts from the infrastructure and create a minimal CloudFormation template for it.

Resources:

    EncryptionKey:
        Type: AWS::KMS::Key
        Properties:
          Description: KMS Key
          KeyPolicy:
            # Allow access from our AWS Account
            Version: "2012-10-17"
            Id: "kms-key"
            Statement:
              - Sid: "Enable IAM User Permissions"
                Effect: Allow
                Principal:
                  AWS:
                    Fn::Join:
                      - ""
                      - - "arn:aws:iam::"
                        - !Ref AWS::AccountId
                        - ":root"
                Action: "kms:*"
                Resource: "*"

    S3BucketFileStorageBucket:
        DependsOn: NewFileNotifications
        Type: AWS::S3::Bucket
        Properties:
          BucketName: randombucketname0928439843293874
          # Enable Encryption
          BucketEncryption:
            ServerSideEncryptionConfiguration:
              - ServerSideEncryptionByDefault:
                  SSEAlgorithm: AES256
          # Block all public access
          AccessControl: Private
          NotificationConfiguration:
            TopicConfigurations:
              - Event: "s3:ObjectCreated:*"
                Topic: !Ref NewFileNotifications
          PublicAccessBlockConfiguration:
            BlockPublicAcls: true
            BlockPublicPolicy: true
            IgnorePublicAcls: true
            RestrictPublicBuckets: true

    NewFileNotifications:
        Type: AWS::SNS::Topic
        Properties:
          KmsMasterKeyId: !Ref EncryptionKey
          TopicName: NewFileNotifications

    NewFileNotificationsTopicPolicy:
        Type: AWS::SNS::TopicPolicy
        Properties:
          PolicyDocument:
            Id: AllowPublishFromS3
            Version: '2012-10-17'
            Statement:
              - Sid: Statement-id
                Effect: Allow
                Principal:
                  AWS: "*"
                Action: sns:Publish
                Resource:
                  Ref: NewFileNotifications
                Condition:
                  ArnLike:
                    aws:SourceArn: 'arn:aws:s3:::randombucketname0928439843293874'
          Topics:
            - Ref: NewFileNotifications

Minimal insofar as CloudFormation templates can be minimal. I deployed it and saw the same behaviour as in the real environment, which was good… and bad. It meant I had done something wrong and couldn’t blame AWS.

You might have noticed something I hadn’t mentioned yet - the bucket and topic were encrypted via KMS. Turns out looking at my own code would have helped - my comment mentions Allow access from our AWS Account but the statement Id says Enable IAM User Permissions. Past me had lied and present me believed him and assumed the account hat access to the key.

This section allows all IAM-Users to access the key:

    EncryptionKey:
        Type: AWS::KMS::Key
        Properties:
          Description: KMS Key
          KeyPolicy:
            # Allow access from our AWS Account
            Version: "2012-10-17"
            Id: "kms-key"
            Statement:
              - Sid: "Enable IAM User Permissions"
                Effect: Allow
                Principal:
                  AWS:
                    Fn::Join:
                      - ""
                      - - "arn:aws:iam::"
                        - !Ref AWS::AccountId
                        - ":root"
                Action: "kms:*"
                Resource: "*"

That’s why the checks S3 does to verify the connection to SNS worked fine when I executed them from the GUI and during CloudFormation deployment. Both are triggered via real Users - in the case of the GUI my own user and the technical user for the pipeline.

After finding out what I had to look for, I quickly found this AWS blog, which describes how to do it properly. Once you know what’s wrong, it seems like the whole internet tells you how to do it properly and makes you feel like an idiot for not knowing it.

After some more research I changed the encryption key configuration to this:

Resources:

    EncryptionKey:
        Type: AWS::KMS::Key
        Properties:
          Description: KMS Key
          KeyPolicy:
            # Allow access from our AWS Account
            Version: "2012-10-17"
            Id: "kms-key"
            Statement:
              - Sid: "Enable IAM User Permissions"
                Effect: Allow
                Principal:
                  AWS:
                    Fn::Join:
                      - ""
                      - - "arn:aws:iam::"
                        - !Ref AWS::AccountId
                        - ":root"
                Action: "kms:*"
                Resource: "*"
              - Sid: "Allow S3 to encrypt"
                Effect: Allow
                Principal:
                  Service: s3.amazonaws.com
                Action:
                 - "kms:GenerateDataKey*"
                 - "kms:Decrypt"
                Resource: "*"
              - Sid: "Allow SNS to encrypt"
                Effect: Allow
                Principal:
                  Service: sns.amazonaws.com
                Action:
                 - "kms:GenerateDataKey*"
                 - "kms:Decrypt"
                Resource: "*"

It allows SNS and S3 to use the KMS Key to encrypt messages.

Summary

If S3 notifications to SNS fail silently, check if you have encryption enabled and make sure that SNS and S3 have access to the key. Out of shame I won’t disclose how long it took me to figure this out ;-)

Similar Posts You Might Enjoy

Enforcing encryption standards on S3-objects

Encrypting objects at rest is a best practice when working with S3. Enforcing this with policies is not as trivial as you may think. There are subtle issues with default encryption, which may result in compliance risks. We’re going to investigate these issues and show you how to solve them. - by Maurice Borgmeier , Gernot Glawe

Build a Serverless S3 Explorer with Dash

Many projects get to the point where your sophisticated infrastructure delivers reports to S3 and now you need a way for your end users to get them. Giving everyone access to the AWS account usually doesn’t work. In this post we’ll look at an alternative - we’re going to build a Serverless S3 Explorer with Dash, Lambda and the API Gateway. - by Maurice Borgmeier

Building an AWS Lambda Telemetry API extension for direct logging to Grafana Loki

In hybrid architectures, serverless functions work together with container solutions. Lambda logs have to be translated when you don`t choose CloudWatch Logs. The old way of doing this is through subscription filters using additional Lambda functions for log transformation. With the Lambda Telemetry API there is a more elegant, performant and cost-effective way. I am using Grafana Loki as a working example and show you how to build a working Lambda-Loki Telemetry APi extension. - by Gernot Glawe