Use Edge Delta to Monitor AWS S3 SQS

Log files in an S3 bucket for Edge Delta to monitor.

3 minute read

Version 2 is no longer supported. This section is for historical reference only.

Overview

This input type allows you to specify log files in an S3 bucket for Edge Delta to monitor. With this action, S3 will be monitored via SQS.

To use this input type, you must have SQS notifications enabled on the target bucket.

Enable SQS Notification

To use this input type, you must have SQS notifications enabled on the target bucket.

Create an SQS queue to receive messages from S3. The SQS queue must have a policy that allows S3 to send messages to the SQS queue.

{
 "Version": "2008-10-17",
 "Id": "__default_policy_ID",
 "Statement": [
   {
     "Sid": "__owner_statement",
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::<account id>:root"
     },
     "Action": "SQS:*",
     "Resource": "arn:aws:sqs:us-west-2:<account id>:"
   },
   {
      "Sid": "s3_send_statement",
      "Effect": "Allow",
      "Principal": {
       "Service": "s3.amazonaws.com"  
      },
      "Action": [
       "SQS:SendMessage"
      ],
      "Resource": "arn:aws:sqs:us-west-2:<account id>:my-sqs",
      "Condition": {
         "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:my-bucket" },
         "StringEquals": { "aws:SourceAccount": "<account id>" }
     }
   }
 ]
}

Configure the S3 bucket to send notifications to the SQS queue.
- To learn how to configure the S3 bucket, review this document from Amazon.
Create a new IAM user with programmatic access type. The Edge Delta agent will use this user to access SQS and S3.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "s3:GetObject",
                "sqs:DeleteMessageBatch",
                "sqs:ReceiveMessage"
            ],
            "Resource": [
                "arn:aws:s3:::my-bucket/*",
                "arn:aws:sqs:us-west-2:<account id>:my-sqs"
            ]
        }
    ]
}

Create an access key for the IAM user.
- You will need the access key ID and secret in the next step.
Add S3 input to the agent.

Example Configuration

  s3_sqs:
    - labels: "errorcheck"
      sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
      access_key_id: "ABCDEFG"
      access_secret: "Dn2djaskl"
      region: "us-west-2"

    - labels: "alb"
      sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/my-alb-logs-sqs"
      access_key_id: "ABCDEFG"
      access_secret: "Dn2djaskl"
      region: "us-west-2"
      log_type: alb

    - labels: "sqs-input-assumes-role"
      sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"
      region: "us-west-2"
      role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
      external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"

Parameters

labels

Required

Enter a descriptive label for this input.

When you create a workflow, you will use this label to enter your input into the workflow.

errorcheckis the default label.

- labels: "errorcheck"

sqs_url

Required

Enter the full URL for the SQS queue.

sqs_url: "https://sqs.us-west-2.amazonaws.com/233765244907/taylan-test-sqs"

access_key_id

Required

Enter the access key ID that the agent will use to pull SQS and download S3 files.

This access key ID must have the following permissions:

sqs:DeleteMessage
sqs:DeleteMessageBatch
sqsReceiveMessage
s3:GetObject

access_key_id: "ABCDEFG"

access_secret

Required

Enter the access secret key that corresponds with the access key ID.

access_secret: "Dn2djaskl"

role_arn

Optional

Enter the ARN that has permissions to use the desired IAM Role

To learn more, review this document from AWS.

role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"

external_id

Optional

Enter the external ID associated with the desired IAM role.

To learn more, review this document from AWS.

external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"

region

Required

Enter the region where the bucket and SQS queue are located.

region: "us-west-2"

log_type

Optional

Enter the desired log types to monitor:

alb
cloudtrail

log_type: alb

filters

Optional

Enter an existing filter to add to this input.

To learn how to create a filter, see Filters.

filters: 
  - info