CloudWatch Pack

Edge Delta Pipeline Pack for CloudWatch

Overview

The CloudWatch Pack ensures ingestion and appropriate processing of AWS CloudWatch data. This pipeline receives logs from the Edge Delta CloudWatch Forwarder Lambda. The forwarder sends logs to the endpoint where each log received contains multiple logs from the same log group. Using this Pack ensures all data is tagged and unrolled appropriately.

Pack Description

Note: The YAML snippets in this section describe the Pack’s internal processing. You do not need to configure these nodes manually. The Pack applies them automatically when you add it to your Cloud pipeline.

1. body_attributes

The Parse JSON node parses the JSON attributes from the body of the message.

- name: body_attributes
  type: parse_json_attributes
  process_field: item.body

2. extract_logEvents

The Extract JSON Field node extracts values from JSON log content in a particular field and uses it as the log’s body field. This node is useful for focusing log analysis on specific pieces of data or for simplifying complex log documents into a more manageable form that retains only the essential information required for further processing and analysis.

- name: extract_logEvents
  type: extract_json_field
  field_path: logEvents.[*]
  keep_log_if_failed: true

3. parse_body_event

The Parse JSON Attributes node parses the JSON attributes from the specified field path in CloudWatch logs and converts them into standalone attributes. This transformation makes accessing and querying individual fields easier.

- name: body_attributes
  type: parse_json_attributes
  process_field: item.body

4. delete_logEvents

The Log Transform node deletes the logEvents attribute that contains the original array of logs.

- name: delete_logEvents
  type: log_transform
  transformations:
  - field_path: attributes.logEvents
    operation: delete

Example Input

Consider the following extract of a CloudWatch log with two events.

Note: Sensitive information has been replaced with dummy data.

{
  "cloud": {
    "resource_id": "arn:aws:lambda:us-east-2:0000:function:EdgeDelta-EdgeDeltaForwarder",
    "account_id": "00000",
    "region": "us-east-2"
  },
  "faas": {
    "name": "example-lambda",
    "version": "$LATEST",
    "request_id": "0000-0000-0000-0000-0000",
    "memory_size": "128",
    "tags": {
      "aws:cloudformation:logical-id": "EdgeDeltaForwarder",
      "aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-2:00000:stack/EdgeDelta-Forwarder-AMD64/0000-0000-0000-0000-0000",
      "aws:cloudformation:stack-name": "EdgeDelta-Forwarder-AMD64",
      "lambda:createdBy": "SAM",
      "serverlessrepo:applicationId": "arn:aws:serverlessrepo:us-west-2:0000:applications/EdgeDelta-Forwarder-AMD64",
      "serverlessrepo:semanticVersion": "0.0.6",
      "tag1": "value1"
    }
  },
  "aws": {
    "log.group.name": "/aws/lambda/example-lambda",
    "log.group.arn": "arn:aws:logs:us-east-2:0000:log-group:/aws/lambda/example-lambda",
    "log.group.tags": {
      "tag2": "value2"
    },
    "log.stream.name": "0000/00/00/[$LATEST]0000",
    "log.message_type": "DATA_MESSAGE",
    "log.subscription_filters": [
      "example-filter"
    ]
  },
  "host.arch": "x86_64",
  "process.runtime.name": "python3.13",
  "logEvents": [
    {
      "id": "00000",
      "timestamp": 0000000000,
      "message": "example-lambda-vlKRPrLfIa\n"
    },
    {
      "id": "00001",
      "timestamp": 0000000000,
      "message": "example-lambda-vlKRPrLfIs\n"
    }
  ]
}

Example Output

Consider the following log emitted from the CloudWatch pack.

Note: This example is not derived from the example above but the input log had the same structure.

It consists of a simple body:

{
  "id": "000000000",
  "timestamp": 000000,
  "message": "example-lambda-vcumAjfzgw"
}

As well as a detailed Attributes field:

{
  "aws.log.group.arn": "arn:aws:logs:us-east-2:0000:log-group:/aws/lambda/example-lambda",
  "aws.log.group.name": "/aws/lambda/example-lambda",
  "aws.log.group.tags.tag2": "value2",
  "aws.log.message_type": "DATA_MESSAGE",
  "aws.log.stream.name": "0000/00/00/[$LATEST]00000",
  "aws.log.subscription_filters.0": "example-filter",
  "cloud.account_id": "0000",
  "cloud.region": "us-east-2",
  "cloud.resource_id": "arn:aws:lambda:us-east-2:0000:function:EdgeDelta-EdgeDeltaForwarder",
  "ed.env.id": "0000-0000-0000-0000-0000",
  "faas.memory_size": "128",
  "faas.name": "example-lambda",
  "faas.request_id": "0000-0000-0000-0000-0000",
  "faas.tags.aws:cloudformation:logical-id": "EdgeDeltaForwarder",
  "faas.tags.aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-2:0000:stack/EdgeDelta-Forwarder-AMD64/0000-0000-0000-0000-0000",
  "faas.tags.aws:cloudformation:stack-name": "EdgeDelta-Forwarder-AMD64",
  "faas.tags.lambda:createdBy": "SAM",
  "faas.tags.serverlessrepo:applicationId": "arn:aws:serverlessrepo:us-west-2:0000:applications/EdgeDelta-Forwarder-AMD64",
  "faas.tags.serverlessrepo:semanticVersion": "0.0.6",
  "faas.tags.tag1": "value1",
  "faas.version": "$LATEST",
  "host.arch": "x86_64",
  "id": "000000",
  "message": "example-lambda-vcumAjfzgw\n",
  "process.runtime.name": "python3.13",
  "timestamp": "0000"
}

Sample Input

{"cloud":{"resource_id":"arn:aws:lambda:us-east-2:000000000000:function:serverlessrepo-EdgeDelta-EdgeDeltaForwarder","account_id":"000000000000","region":"us-east-2"},"faas":{"name":"example-lambda","version":"$LATEST","request_id":"23bfe1be-a184-43bb-9ea5-fb52d33d61d4","memory_size":"128","tags":{"aws:cloudformation:logical-id":"EdgeDeltaForwarder","aws:cloudformation:stack-id":"arn:aws:cloudformation:us-east-2:000000000000:stack/serverlessrepo-EdgeDelta-Forwarder-AMD64/b234f090-a5f1-11ef-bc4a-0a90a7175c75","aws:cloudformation:stack-name":"serverlessrepo-EdgeDelta-Forwarder-AMD64","lambda:createdBy":"SAM","serverlessrepo:applicationId":"arn:aws:serverlessrepo:us-west-2:233765244907:applications/EdgeDelta-Forwarder-AMD64","serverlessrepo:semanticVersion":"0.0.6","tag1":"value1"}},"aws":{"log.group.name":"/aws/lambda/example-lambda","log.group.arn":"arn:aws:logs:us-east-2:000000000000:log-group:/aws/lambda/example-lambda","log.group.tags":{"tag2":"value2"},"log.stream.name":"2024/11/19/[$LATEST]e3199c8fcb374f978e12a020acc78362","log.message_type":"DATA_MESSAGE","log.subscription_filters":["cw1"]},"host.arch":"x86_64","process.runtime.name":"python3.13","logEvents":[{"id":"38626090287414021239190390440846764370530496153856770048","timestamp":1732053792084,"message":"INIT_START Runtime Version: python:3.13.v13\tRuntime Version ARN: arn:aws:lambda:us-east-2::runtime:b881cbc9a10a8bcb3def9d9e9fe38f922bb36510a1d92d4ce85cf2a899eeabd8\n"},{"id":"38626090289554892778249330262434193324704738858430889985","timestamp":1732053792180,"message":"START RequestId: 0cfd4e3c-cc4c-4bd7-8cb1-e2901fbb629b Version: $LATEST\n"},{"id":"38626090289577193523447860885575729042977387219936870402","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ\n"},{"id":"38626090289577193523447860885575729042977387219936870403","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181543315\n"},{"id":"38626090289577193523447860885575729042977387219936870404","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181554207\n"},{"id":"38626090289577193523447860885575729042977387219936870405","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181562744\n"},{"id":"38626090289577193523447860885575729042977387219936870406","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181568639\n"},{"id":"38626090289577193523447860885575729042977387219936870407","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181574403\n"},{"id":"38626090289577193523447860885575729042977387219936870408","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181581023\n"},{"id":"38626090289577193523447860885575729042977387219936870409","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181587230\n"},{"id":"38626090289577193523447860885575729042977387219936870410","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181592960\n"},{"id":"38626090289577193523447860885575729042977387219936870411","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181598519\n"},{"id":"38626090289577193523447860885575729042977387219936870412","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181605151\n"},{"id":"38626090289644095759043452755000336197795332304454811661","timestamp":1732053792184,"message":"END RequestId: 0cfd4e3c-cc4c-4bd7-8cb1-e2901fbb629b\n"},{"id":"38626090289644095759043452755000336197795332304454811662","timestamp":1732053792184,"message":"REPORT RequestId: 0cfd4e3c-cc4c-4bd7-8cb1-e2901fbb629b\tDuration: 2.06 ms\tBilled Duration: 3 ms\tMemory Size: 128 MB\tMax Memory Used: 31 MB\tInit Duration: 93.73 ms\t\n"}]}