Connecting S3 with Lambda on AWS CDK in Typescript [VIDEO]

Let’s revisit our pipeline once again.

We want to send notifications from S3 to our Lambda whenever we put a file into our S3 bucket, and in this tutorial, we are using AWS CDK in Typescript to achieve that.

1. Ensure logging within the AWS Lambda

This is important to prove that the Lambda successfully receives the S3 notification from the bucket, and we can trace it within AWS Cloudwatch.

import logging
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)

def lambda_handler(event, context):
    # Outputs the incoming event into CW logs
    logger.info(event)
    
    return {
        "statusCode":200
    }
# For direct invocation and testing on the local machine
if __name__ == '__main__':
    print(lambda_handler(None,None))

Otherwise, there is no way we would know if our pipeline is working.

2. Exporting the Lambda function as an object

For the S3 notifications to work, the S3 bucket needs to know which lambda function to send its notifications over.
This is one of the main strengths of AWS CDK where we can export our Lambda function as an object for other infrastructures to leverage on.

In lib/basic-lambda-stack.ts:

import * as cdk from '@aws-cdk/core';
import * as lambda from '@aws-cdk/aws-lambda';
export class basicLambdaStack extends cdk.Stack{
  // Making the object accessible for reuseability
  public readonly lambdaFunction: lambda.Function;

  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    const function_name = 'gefyra-basic-lambda';
    const lambda_path = 'src/lambda/basic_lambda';

    // Initialization of the lambda function
    this.lambdaFunction = new lambda.Function(this, function_name, {
        functionName: function_name,
        runtime: lambda.Runtime.PYTHON_3_8,
        code: lambda.Code.fromAsset(lambda_path),
        handler: "lambda_function.lambda_handler"
    });
  }
}

As you can see:

• Within the class itself, we would expose an object via public readonly lambdaFunction: lambda.Function;
• Use the same object to initialize the Lambda function.

From the above code snippet, we allow our Lambda function to be reused within the stack.

3. Get the S3 bucket ready for the Lambda

Install @aws-cdk/aws-s3-notifications with npm install @aws-cdk/aws-s3-notifications. Make sure to update all your AWS CDK libraries at the same time to avoid conflicts and deployment errors.

We are going to modify the lib/s3-bucket-stack.ts to receive the Lambda object and attribute S3 event notifications to the Lambda.

import * as cdk from '@aws-cdk/core';
import * as s3 from "@aws-cdk/aws-s3";
import * as s3n from '@aws-cdk/aws-s3-notifications';
import * as lambda from '@aws-cdk/aws-lambda';

// Allows the stack to receive a lambda.function object
export interface S3Props extends cdk.StackProps{
  readonly lambdaFunction: lambda.Function;
}

export class S3BucketStack extends cdk.Stack {
  public readonly bucket: s3.Bucket;

  // Replace the props with our new interface
  constructor(scope: cdk.Construct, id: string, props: S3Props) {
    super(scope, id, props);

    // The code that defines your stack goes here
    this.bucket = new s3.Bucket(this, "gefyra-data-collection-dev",{
      versioned: false,
      bucketName: "gefyra-data-collection-dev",
      publicReadAccess: false,
      blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    });

    // Assigning notifications to be sent to the Lambda function
    this.bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(props.lambdaFunction));
  }
}  

So let’s break this down one by one.

Creating an interface to receive the Lambda function object

// Allows the stack to receive a lambda.function object
export interface S3Props extends cdk.StackProps{
  readonly lambdaFunction: lambda.Function;
}

The above code snippet creates a dependency between the Lambda stack and the S3 stack. This also breaks the bin/gefyra-cdk-demo.ts script where we run our code to deploy the infrastructure because it is now expecting a lambdaFunction object as input.

We will rectify it later.

Assign the newly-created interface as part of the constructor

  // Replace the props with our new interface
  constructor(scope: cdk.Construct, id: string, props: S3Props){

Typescript is strongly-typed, and when we assign our interface as part of the constructor, the props object will not accept any objects that are foreign to the interface.

This reduces human error where a developer might wrongly attribute an incorrect class or object to the variable.

Attribute the S3 notifications to the Lambda function.

    // Assigning notifications to be sent to the Lambda function
    this.bucket.addEventNotification(s3.EventType.OBJECT_CREATED, new s3n.LambdaDestination(props.lambdaFunction));

With the s3.EventType.OBJECT_CREATED, every time a user puts a file within the bucket, it would send a notification to the Lambda function that we assign to above.

Also, you can fine-tune your event notifications with s3.EventType.OBJECT_REMOVED, or any other configuration based on your use case. We will only use this generic event type here.

We don’t have to worry about IAM permissions between the S3 bucket and the Lambda because AWS CDK would do that for us. That’s the beauty of AWS CDK, and it removes the complexity of finding out the necessary permissions between resources.

However, fine-tuning permissions is still the best practice for security.

4. Prepare the executable script for deployment

In the bin/gefyra-cdk-demo.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '@aws-cdk/core';
//import { GefyraCdkDemoStack } from '../lib/gefyra-cdk-demo-stack';
import { S3BucketStack } from '../lib/s3-bucket-stack';
import { basicLambdaStack } from '../lib/basic-lambda-stack';

const app = new cdk.App();
//new GefyraCdkDemoStack(app, 'GefyraCdkDemoStack');

// Deploying basic Lambda function
const basic_lambda_stack = new basicLambdaStack(app, 'basicLambdaStack');

// Creating an S3 bucket stack
const s3_bucket_stack = new S3BucketStack(app, 'gefyraS3Stack', {
  lambdaFunction: basic_lambda_stack.lambdaFunction
});

// Re-using assets
const bucket = s3_bucket_stack.bucket;

As previously indicated, all files within the lib folder are scripts that initialize the infrastructure, and all files within the bin folder execute the deployment.

Nothing much changed with the basic_lambda_stack object. However, you’ll realize that the s3_bucket_stack now requires a Lambda function object as input. Typescript will throw an error if you missed that out.

We can now deploy our infrastructure.

5. Deploying the pipeline

When you run npm run build && cdk synth, you’ll face a decision of what to deploy first.

If you’re don’t know AWS CDK, you would be a little worried because the order of operations could mess up deployment sequences. For S3 notifications to work, it would need an existing Lambda ARN to use.

However, AWS CDK doesn’t care. If we deploy the S3 bucket first, AWS CDK is smart enough to deploy the Lambda function first before deploying the S3 bucket.

And if you deploy the Lambda function first, it wouldn’t create conflicts if we deploy the S3 bucket separately.
We can see visual proof that the S3 bucket has successfully linked itself to the Lambda function:

6. Testing the pipeline with a file.

1. Drop a file into the S3 bucket.
2. Check the Cloudwatch logs for that Lambda function to see if the notification was successful.

You should be able to see that the Lambda function receives a notification from S3 whenever a file was dropped into the bucket.

Pro-Tip

It’s a hassle to keep testing the Lambda by putting files into the S3 bucket all the time if we know it was going to work.

Saving the S3 notification into a JSON file could ease testing practices, but the Cloudwatch log uses single quotes instead of double quotes within the payload. This doesn’t allow us to save the payload by copying and pasting.

There is a quick and simple way to fix that within your browser:

1. Open the console of your browser (I tested with Chrome and Firefox).
2. Type console.log(JSON.stringify(<copy and paste the S3 notification here>));

It would return a proper JSON string where you can save it as a file or reuse it as a test case within the AWS console.

When we create a tutorial for unit and/or integration tests in the future, we can mock the payload with the same JSON file.

Git Repo Reference

All the above code can be found in my repo: https://github.com/jonathan-moo/gefyra-cdk-demo/tree/CDK-T4