How to filter S3 events by object size

[cloudonaut]

While answering a support request for bucketAV, I stumbled upon the following question:

Is there a way to only scan S3 objects with a size of less than 1 GB for malware?

This translates to the more general question:

How to filter S3 events by object size?

Filtering S3 events by object size is helpful in the following scenarios:

• How to get notified via email when someone uploads a large file to S3?
• How to ensure only files smaller than 100 MB are processed by Lambda to avoid timeouts?
• How to trigger an ECS task after an archive with more than 1 GB has been uploaded to S3?

Luckily, there are simple ways to filter S3 events by object size.

/images/2023/02/how-to-filter-s3-events-by-object-size-title.jpg

S3 Event Notifications vs. EventBridge

Nowadays, there are two options to react to new or modified S3 objects:

• S3 Event Notifications has been around for years and allows us to send events to SNS, SQS, and Lambda.
• EventBridge, the serverless event bus, is the state-of-the-art approach for building event-driven systems on AWS.

Both options allow you to filter events based on the S3 object size.

Filtering S3 Event Notifications by object size

Assuming you configured S3 Event Notifications to deliver events to an SNS topic. The following filter policy only delivers events about an object with an object size of less than 1000000000 bytes (1 GB) to the subscriber.

{ "Records": { "s3": { "object": { "size": [{"numeric": ["<", 1000000000]}] } } } }

First, create an SNS topic.

Second, configure S3 Event Notifications, as illustrated in the following screenshots.

/images/2023/02/s3-events-sns-01.png

/images/2023/02/s3-events-sns-02.png

Third, create a subscription for the SNS topic.

/images/2023/02/s3-events-sns-03.png

Apply the subscription filter as shown in the following screenshot. Make sure to select the policy scope MessageBody.

/images/2023/02/s3-events-sns-04.png

Filtering EventBridge events by S3 object size

After enabling EventBridge events, the following event pattern matches events about new or modified objects with a size of less than 1000000000 bytes (1 GB).

{ "source": ["aws.s3"], "detail-type": ["Object Created"], "detail": { "object": { "size": [{ "numeric": ["<", 1000000000] }] } } }

First, enable EventBridge events for the S3 bucket, as illustrated in the following screenshot.

/images/2023/02/s3-events-eventbridge-01.png

Second, create an EventBridge rule, as shown in the following screenshot.

/images/2023/02/s3-events-eventbridge-02.png

Third, copy and paste the event pattern as demonstrated in the following screenshot.

/images/2023/02/s3-events-eventbridge-03.png

Fourth, create the EventBridge rule.

Summary

Both S3 Event Notifications and EventBridge events allow you to filter events about new or modified S3 events by object size.

View the full article