Edge Delta AWS CloudWatch Log Events

AWS CloudWatch Log Events.

4 minute read

Overview

This input type allows you to specify a set of AWS CloudWatch Log Events for Edge Delta to monitor. With this input, you can monitor multiple regions and log streams.

Example

cloudwatches:
    - labels: "cloudwatch-input-with-rate-limiters"
      region: ".*"
      log_group: "^/ed-log"
      log_stream: ".*"
      interval: 5m    
      rate_limiters:
        - api_name: "DescribeLogGroups"
          every: 1s
          limit: 4
        - api_name: "DescribeLogStreams"
          every: 1s
          limit: 4
        - api_name: "GetLogEvents"
          every: 1s
          limit: 22
    - labels: "us-west-2_ed-log-group_admin-api"
      region: "^us.*$"
      log_group: /ed-log-group
      log_stream: "^log.*$"
      lookback: 1h
      interval: 1m
      prepend_timestamp: true
      result_limit: 5000
    - labels: "us_ed-log-group_admin"
      region: "^us.*$"
      log_group: "/ed-log-group"
      log_stream: "^admin.*$"
      interval: 5m
    - labels: "ed-log-group"
      region: ".*"
      log_group: "/ed-log-group"
      log_stream: ".*"
      interval: 5m
    - labels: "ed-log-with-regex-group-name"
      region: ".*"
      log_group: "^/ed-log"
      log_stream: ".*"
      interval: 5m
    - labels: "cloudwatch-input-assumes-role"
      region: "us-west-2"
      log_group: "/ed-log-group"
      log_stream: ".*"
      interval: 5m
      role_arn: "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
      external_id: "053cf606-8e80-47bf-b849-8cd1cc826cfc"
    - labels: "cloudwatch-input-host-filtered"
      host: "myhost"
      region: ".*"
      log_group: "^/ed-log"
      log_stream: ".*"
      interval: 5m
    - labels: "cloudwatch-input-with-aws-creds"
      region: ".*"
      log_group: "^/ed-log"
      log_stream: ".*"
      interval: 5m
      aws_key_id: '{{ Env "AWS_KEY_ID" }}'
      aws_sec_key: '{{ Env "AWS_SECRET_KEY" }}'

Required Parameters

labels

The labels parameter specifies one or more names for the input. You refer to one of these labels in other places, for example to refer to a specific input in a workflow. Labels must be unique within the inputs section. It is a yaml list element so it begins with a - and a space followed by the string. A label is required for an input.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"

log_group

The log_group parameter specifies the CloudWatch Logs agent. It is specified with a Golang regex expression. A log_group is required for a CloudWatch input.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"

Optional Parameters

filters

Enter an existing filter to add to this input. To learn more, see Filters.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      filters:
      - info

interval

Enter an internal rate to look for new, incoming log events. By default, this parameter is set to 1 minute.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      interval: 2m

log_stream

Enter a log stream pattern via a Golang regex expression.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      log_stream: "<regex>"

lookback

Enter an internal rate to monitor past data. By default, this parameter is set to 1 hour.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      lookback: 2h

prepend_timestamp

Enter true or false to add an event timestamp as a prefix to the event message with a tab ("\t") delimiter.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      prepend_timestamp: true

rate_limiters

Specify the API rate limit to conform with the quotas set by Amazon Web Services. Specify the CloudWatch resource using the api_name. Specify the time frame with the every parameter and the number of transactions with the limit parameter.

For example, the current quota from CloudWatch for DescribeLogGroups is 5 per second so an agent limit of 4 transactions every 1 second will fall under the quota.

If you have multiple CloudWatch inputs configured you need to split your quota between the inputs. For example, if you have two inputs using the DescribeLogGroups resource they should each be limited to half the total quota: 2 transactions per second each to fall under the 5 transactions per second total quota.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      rate_limiters:
        - api_name: "DescribeLogGroups"
          every: 1s
          limit: 4
        - api_name: "DescribeLogStreams"
          every: 1s
          limit: 4
        - api_name: "GetLogEvents"
          every: 1s
          limit: 22

region

You can define a region pattern via Golang regex expression. For example, for all regions in United States, enter: region: "^us.*$"

By default, an AWS account is not enabled with all regions. As a result, you can monitor AWS CloudWatch Log Events for all regions without defining a region in the config file; the Edge Delta Agent will obtain and monitor logs from all enabled regions in your account. To accomplish this, you must add the ec2:DescribeRegions permission to your account. To learn more, review the AWS document about DescribeRegions.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      region: "<regex>"

result_limit

Enter a limit for the the maximum number of log events that should be returned. By default, this parameter will return up to 1 MB of log events, which can be up to 10,000 log events.

inputs:
  cloudwatches:
    - labels: "<input name>"
      log_group: "<regex>"
      result_limit: 5000