Use the CDK to trigger your Lambda function in sub-minute intervals

Lambda functions are everywhere in AWS. One of the many use cases is to periodically perform some action. This can mean starting or stopping instances or fetching data from an API and storing it somewhere. Especially in the latter case it’s useful to be able to query the API every couple of seconds. Unfortunately this isn’t possible using the native CloudWatch events trigger. In this post I’ll show you an alternative.

Why is this not possible natively? The common mechanism that’s used to schedule Lambda functions at certain intervals or point in times is through CloudWatch Events aka. Event Bridge. The system allows you to create interval-based or cron-based rules that can notify other AWS services. Unfortunately the smallest time-resolution either of these systems support is one minute.

You could of course use the Event Bridge trigger to run the function every minute and then use sleep() in your code to perform the operation every couple of seconds. That has several drawbacks though. You would create a long-running Lambda function that does nothing most of the time. Also, if a single fetch-operation takes longer than your interval, your timing for the next iteration will be messed up. By this point your code wouldn’t be only event based anymore, you’d effectively have to implement a scheduler in your Lambda function. Fortunately there are better options.

How can we solve this problem? We can use a combination of Event Bridge Rules and Step Function state machines to periodically trigger our Lambda functions. The diagram below shows the implementation I chose for this problem. You can think of this approach as two loops.

Architecture

The outer “loop” is handled by Event Bridge. Every minute a rule triggers the inner loop - a step function. This step function is set up in a way to handle the operations for a given minute. It will trigger Lambda functions asynchronously using the “Event” invocation type and then sleep until the next interval. This is a fairly simple setup and has the benefit that it’s also inexpensive.

The code for this solution is written in Python and available on Github. In the sub_minute_lambda_trigger/infrastructure.py you’ll find the following construct. It allows you to wrap a Lambda function with a construct in order to facilitate the interval trigger.

class SubMinuteLambdaTrigger(Construct):

    def __init__(
        self,
        scope: Construct,
        construct_id: str,
        interval: int,
        lambda_function: _lambda.Function,
        enabled=True,
         **kwargs
    ):
        super().__init__(scope, construct_id, **kwargs)
		# Full code on Github...

To use it you create an instance of the construct, pass the reference to your Lambda function to it and tell it at which interval to run the function. The rest is handled automatically by the construct. It will set up the event rule, create and configure the step function and invoke your function on time.

# Create SubMinuteLambdaTrigger with Lambda Function and interval
SubMinuteLambdaTrigger(
	self,
	"sub-minute-trigger",
	interval=10,
	lambda_function=lambda_function,
	enabled=True,
)

There are, however, some caveats here. This solution creates a standard step function and for most purposes they will be fast/accurate enough. If you require a tighter timing, you might want to change the type of the step function to express. This will however increase your costs!

The second caveat is that this construct will only accept intervals of which 60 is a multiple, i.e. 60 % interval = 0, so numbers like 2, 5, 10, 12, 15, 20, 30 will be accepted and numbers such as 7, 13 or anything above 30 will be rejected. This makes the whole system more accurate, because we can only rely on the outer loop triggering the state machine every minute and can’t communicate an offset.

You can extend it to allow different intervals, but there is a trade off. If you want to disable this kind of trigger, you can disable the rule that triggers the step function. Given this setup, the trigger will operate for at most one more minute before it ceases invocations. If you extend the construct to allow different intervals you’d have to decrease the number of Event Bridge invocations. That means the step function will run for a longer time and it takes more time until the invocations are stopped.

Summary

In this post I shared with you how you can use step functions and Event Bridge in combination to achieve sub-minute interval triggers for Lambda functions. This can be implemented using the CDK and the code in this repository.

Thank you for reading and I hope you liked this post. If there are any questions or concerns, feel free to reach out to me through the social media channels listed in my bio.

— Maurice

Similar Posts You Might Enjoy

Enhance Lambda Security with new Amazon Inspector Vulnerability Management and prevent 'log4jgate'

Detect the crack in the window (or the lambda library) before it breaks: As we have seen during the last month, also well known libraries like log4j can have previously unknown vulnerabilities. Therefore scanning your Lambda application before deploying is not enough! What happens if a new cybersecurity vulnerability occurs while your functions are running? Solution: Amazon Inspector. Only problem: it`s not available for Lambda yet. Fortunately, you can deploy Lambda with container images and Inspector will continuously scan these images for you. Want to know how set up Lambda & Inspector and see how evil Node vulnerabilities are detected? Read on! - by Gernot Glawe

The CDK Book: The missing Go Code Examples

The CDK Book The CDK Book “A Comprehensive Guide to the AWS Cloud Development Kit” is a book by Sathyajith Bhat, Matthew Bonig, Matt Coulter, Thorsten Hoeger written end of 2021. Because the CDK itself is polyglott with jsii, the TypeScript examples are automatically translated in other languages. So the example CDK code used in the book is jsii generated, and there are samples for TypeScript, Python, Java and C#. - by Gernot Glawe

Views of the Pyramids: From a monolithic Test process to a Serverless Test Automation with CodeBuild

Comparing the development methodology of a monolithic program to a Serverless IAC application you will see that the power of DevOps lies in automating everything. I will show you a working example of a serverless CI pipeline with automated unit, integration and end2end test and test reports in CodeBuild. The full source is written GO, with references to Node.JS and python for the test parts. - by Gernot Glawe