Oftentimes you may need to process only the new files on an AWS Simple Storage Service (S3), but the problem lies in identifying these files. The solution: Simple Queue Service, or SQS.

This walkthrough assumes an S3 bucket has already been created.

Configure SQS

First, identify your S3 bucket's region via the S3 console:

Screen-Shot-2019-02-25-at-20.05.10

The region is important because the SQS instance needs to reside within the same region as the S3 bucket.

Next, find the bucket's ARN, or Amazon Resource Name, by clicking on the bucket's row and selecting Copy Bucket ARN:

Screen-Shot-2019-02-25-at-20.20.36

Once you have identified the ARN and region, create a SQS service via the AWS Console in that region. During the queue creation process you will prompted to provide a name for the queue and choose between a Standard Queue and FIFO Queue. The queue's name is inconsequential but you must select Standard Queue; FIFO Queues are not supported by S3 events. Once you're done, select Quick-Create Queue.

Once the queue is created right-click and select Configure Queue to edit its settings per your project's requirements, but note none of the configuration items will prevent event notification delivery.

Next, when your SQS is selected on the console several tabs will apear at the bottom of the browser. Click the Permissions tab and select Edit Policy Document (Advanced). Below is an example of an IAM policy that you attach to the destination SQS queue:

{
 "Version": "2008-10-17",
 "Id": "example-ID",
 "Statement": [
  {
   "Sid": "example-statement-ID",
   "Effect": "Allow",
   "Principal": {
     "AWS": "*"  
   },
   "Action": [
    "SQS:SendMessage"
   ],
   "Resource": "SQS-ARN",
   "Condition": {
      "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:bucket-name" }
   }
  }
 ]
}

Configure S3 Events

Go back to the S3 console and click on your bucket's name.

Select Properties:
Screen-Shot-2019-02-25-at-20.46.39

Then scroll down to the Advanced section and select Events:
Screen-Shot-2019-02-25-at-20.47.18

In Name, type a descriptive name for your event configuration. If you do not enter a name, a GUID is autogenerated and used for the name.

Under Events, select one or more of the type of event occurrences that you want to receive notifications for. When the event occurs a notification is sent to the SQS destination we configured in the first part of this walkthrough. For a description of the event types, see Amazon S3 Event Notification Types and Destinations. Select All object create events to trigger a notification when a new file is added.

Type an object name Prefix and/or a Suffix to filter the event notifications by the prefix and/or suffix. For example, you can set up a filter so that you are sent a notification only when files are added to an image folder (objects with the name prefix images/).

For Send To select SQS Queue and choose the name of the SQS Queue you created earlier. Click Save and voila! Now when your specified event occurs you will receive a message in your Queue.

You can view your queue's messages in your browser by navigating to the queue's deployment on the AWS console, right-clicking on the queue, and selecting View/Delete Messages.