Edge Delta Amazon CloudWatch Destination

Send logs to Amazon CloudWatch.

5 minute read

Overview

The Amazon CloudWatch destination node send data to Amazon CloudWatch.

incoming_data_types: cluster_pattern_and_sample, log, metric, custom

Example Configuration

In this configuration, expressions are used to dynamically determine the values for namespace, group_name, and stream_name based on the attributes of the data item. The resulting values determine where data items will be sent.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    namespace_expression: 'has(item.attributes) && has(item.attributes.namespace) && item.attributes.namespace != "" ? item.attributes.namespace : "default_namespace"'
    group_name_expression: 'has(item.attributes) && has(item.attributes.group) && item.attributes.group != "" ? item.attributes.group : "default_group"'
    stream_name_expression: 'has(item.attributes) && has(item.attributes.stream) && item.attributes.stream != "" ? item.attributes.stream : "default_stream"'
    create_missing_group_and_stream: true

namespace_expression: This expression checks if the item has an attributes field and if the attributes field contains a namespace key. If item.attributes.namespace exists and is not an empty string, the expression evaluates to item.attributes.namespace. If item.attributes.namespace does not exist or is an empty string, the expression defaults to "default_namespace".
group_name_expression: Similar to namespace_expression, this expression checks for the presence of a group key within item.attributes. If item.attributes.group exists and is not an empty string, it returns item.attributes.group. If item.attributes.group is missing or empty, it defaults to "default_group".
stream_name_expression: This expression follows the same logic pattern as the previous two, but it checks for a stream key within item.attributes. If item.attributes.stream is present and not empty, the expression evaluates to item.attributes.stream. If item.attributes.stream is absent or empty, it defaults to "default_stream".

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: aws_cloudwatch_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

region

The region parameter specifies the region where the CloudWatch resource is. It is specified as a string and is required.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2

Optional Parameters

At least one authentication method is required.

aws_key_id

The aws_key_id parameter is the AWS key ID that has permission to access CloudWatch. It is used with aws_sec_key. It is specified as a string and is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2

aws_sec_key

The aws_sec_key parameter is the AWS secret for the key ID that has permission to access CloudWatch. It is used with aws_key_id. It is specified as a string and is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2

role_arn

The role_arn parameter is used if authentication and authorization is performed using an assumed AWS IAM role. It should consist of the account ID and role name. A role_arn is optional depending on the access configuration.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    role_arn: <role ARN>
    region: us-east-2

external_id

The external_id parameter is a unique identifier to avoid a confused deputy attack. It is specified as a string and is optional. While external_id is optional, when configured it must be used with role_arn

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    role_arn: <role ARN>
    external_id: <ID>
    region: us-east-2

namespace_expression

The namespace_expression parameter is used to specify an expression to dynamically determine which namespace incoming metrics will be sent to. It is specified as a string and is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    namespace_expression: 'has(item.attributes) && has(item.attributes.namespace) && item.attributes.namespace != "" ? item.attributes.namespace : "default_namespace"'

group_name_expression

The group_name_expression parameter is used to specify an expression to dynamically determine which group incoming logs and patterns will be sent to. It is specified as a string and is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    group_name_expression: 'has(item.attributes) && has(item.attributes.group) && item.attributes.group != "" ? item.attributes.group : "default_group"'

stream_name_expression

The stream_name_expression parameter is used to specify an expression to dynamically determine which stream incoming logs and patterns will be sent to. It is specified as a string and is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    stream_name_expression: 'has(item.attributes) && has(item.attributes.stream) && item.attributes.stream != "" ? item.attributes.stream : "default_stream"'

create_missing_group_and_stream

The create_missing_group_and_stream parameter is used create missing group and/or streams on Cloudwatch if they don’t exist. It is specified as a Boolean and is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    group_name_expression: 'has(item.attributes) && has(item.attributes.group) && item.attributes.group != "" ? item.attributes.group : "default_group"'
    stream_name_expression: 'has(item.attributes) && has(item.attributes.stream) && item.attributes.stream != "" ? item.attributes.stream : "default_stream"'
    create_missing_group_and_stream: true

buffer_ttl

The buffer_ttl parameter configures the time-to-Live for unsuccessful items, which indicates when to discard them. It is specified as a duration, has a default of 10m, and it is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    buffer_ttl: 20m

buffer_path

The buffer_path parameter configures the path to store unsuccessful items. Unsuccessful items are stored there to be retried back (exactly once delivery). It is specified as a string and it is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    buffer_path: <path to unsuccessful items folder>

buffer_max_bytesize

The buffer_max_bytesize parameter configures the maximum byte size for total unsuccessful items. If the limit is reached, the remaining items are discarded until the buffer space becomes available. It is specified as a datasize.Size, has a default of 0 indicating no size limit, and it is optional.

  - name: my_cloudwatch_output
    type: aws_cloudwatch_output
    aws_key_id: abc
    aws_sec_key: xyz
    region: us-east-2
    buffer_max_bytesize: 2048