CloudWatch Pack

Edge Delta Pipeline Pack for CloudWatch

Overview

The CloudWatch pack ensures ingestion and appropriate processing of AWS CloudWatch data. This pipeline recieves the logs from the Edge Delta CloudWatch Forwarder Lambda. The Edge Delta CloudWatch Forwarder Lambda sends logs to the endpoint where each log recieved contains multiple logs from the same log group. Using this pack ensures all data is tagged and unrolled appropriately. There is prerequisite to add the (Edge Delta CloudWatch Forwader)[https://docs.edgedelta.com/aws-lambda-forwarder/].

Pack Description

1. body_attributes

The Parse JSON node parses the JSON attributes from the body of the message.

- name: body_attributes
  type: parse_json_attributes
  process_field: item.body

2. extract_logEvents

The Extrace JSON Field node extracts values from JSON log content in a particular field and uses it as the log’s body field. This node is useful for focusing log analysis on specific pieces of data or for simplifying complex log documents into a more manageable form that retains only the essential information required for further processing and analysis.

- name: extract_logEvents
  type: extract_json_field
  field_path: logEvents.[*]
  keep_log_if_failed: true

3. parse_body_event

The Parse JSON Attributes node parses the JSON attributes from the specified field path in CloudWatch logs and converts them into standalone attributes. This transformation makes accessing and querying individual fields easier.

- name: body_attributes
  type: parse_json_attributes
  process_field: item.body

4. delete_logEvents

The Log Transform node deletes the logEvents attributes that contains all of the logs sent down

- name: delete_logEvents
  type: log_transform
  transformations:
  - field_path: attributes.logEvents
    operation: delete

Example Input

Consider the following extract of a CloudWatch log with two events.

Note: Sensitive information has been replaced with dummy data.

{
  "cloud": {
    "resource_id": "arn:aws:lambda:us-east-2:0000:function:EdgeDelta-EdgeDeltaForwarder",
    "account_id": "00000",
    "region": "us-east-2"
  },
  "faas": {
    "name": "example-lambda",
    "version": "$LATEST",
    "request_id": "0000-0000-0000-0000-0000",
    "memory_size": "128",
    "tags": {
      "aws:cloudformation:logical-id": "EdgeDeltaForwarder",
      "aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-2:00000:stack/EdgeDelta-Forwarder-AMD64/0000-0000-0000-0000-0000",
      "aws:cloudformation:stack-name": "EdgeDelta-Forwarder-AMD64",
      "lambda:createdBy": "SAM",
      "serverlessrepo:applicationId": "arn:aws:serverlessrepo:us-west-2:0000:applications/EdgeDelta-Forwarder-AMD64",
      "serverlessrepo:semanticVersion": "0.0.6",
      "tag1": "value1"
    }
  },
  "aws": {
    "log.group.name": "/aws/lambda/example-lambda",
    "log.group.arn": "arn:aws:logs:us-east-2:0000:log-group:/aws/lambda/example-lambda",
    "log.group.tags": {
      "tag2": "value2"
    },
    "log.stream.name": "0000/00/00/[$LATEST]0000",
    "log.message_type": "DATA_MESSAGE",
    "log.subscription_filters": [
      "example-filter"
    ]
  },
  "host.arch": "x86_64",
  "process.runtime.name": "python3.13",
  "logEvents": [
    {
      "id": "00000",
      "timestamp": 0000000000,
      "message": "example-lambda-vlKRPrLfIa\n"
    },
    {
      "id": "00001",
      "timestamp": 0000000000,
      "message": "example-lambda-vlKRPrLfIs\n"
    }
  ]
}

Example Output

Consider the following log emitted from the CloudWatch pack.

Note: This example is not derived from the example above but the input log had the same structure.

It consists of a simple body:

{
  "id": "000000000",
  "timestamp": 000000,
  "message": "example-lambda-vcumAjfzgw"
}

As well as a detailed Attributes field:

{
  "aws.log.group.arn": "arn:aws:logs:us-east-2:0000:log-group:/aws/lambda/example-lambda",
  "aws.log.group.name": "/aws/lambda/example-lambda",
  "aws.log.group.tags.tag2": "value2",
  "aws.log.message_type": "DATA_MESSAGE",
  "aws.log.stream.name": "0000/00/00/[$LATEST]00000",
  "aws.log.subscription_filters.0": "example-filter",
  "cloud.account_id": "0000",
  "cloud.region": "us-east-2",
  "cloud.resource_id": "arn:aws:lambda:us-east-2:0000:function:EdgeDelta-EdgeDeltaForwarder",
  "ed.env.id": "0000-0000-0000-0000-0000",
  "faas.memory_size": "128",
  "faas.name": "example-lambda",
  "faas.request_id": "0000-0000-0000-0000-0000",
  "faas.tags.aws:cloudformation:logical-id": "EdgeDeltaForwarder",
  "faas.tags.aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-2:0000:stack/EdgeDelta-Forwarder-AMD64/0000-0000-0000-0000-0000",
  "faas.tags.aws:cloudformation:stack-name": "EdgeDelta-Forwarder-AMD64",
  "faas.tags.lambda:createdBy": "SAM",
  "faas.tags.serverlessrepo:applicationId": "arn:aws:serverlessrepo:us-west-2:0000:applications/EdgeDelta-Forwarder-AMD64",
  "faas.tags.serverlessrepo:semanticVersion": "0.0.6",
  "faas.tags.tag1": "value1",
  "faas.version": "$LATEST",
  "host.arch": "x86_64",
  "id": "000000",
  "message": "example-lambda-vcumAjfzgw\n",
  "process.runtime.name": "python3.13",
  "timestamp": "0000"
}

Sample Input

{"cloud":{"resource_id":"arn:aws:lambda:us-east-2:000000000000:function:serverlessrepo-EdgeDelta-EdgeDeltaForwarder","account_id":"000000000000","region":"us-east-2"},"faas":{"name":"example-lambda","version":"$LATEST","request_id":"23bfe1be-a184-43bb-9ea5-fb52d33d61d4","memory_size":"128","tags":{"aws:cloudformation:logical-id":"EdgeDeltaForwarder","aws:cloudformation:stack-id":"arn:aws:cloudformation:us-east-2:000000000000:stack/serverlessrepo-EdgeDelta-Forwarder-AMD64/b234f090-a5f1-11ef-bc4a-0a90a7175c75","aws:cloudformation:stack-name":"serverlessrepo-EdgeDelta-Forwarder-AMD64","lambda:createdBy":"SAM","serverlessrepo:applicationId":"arn:aws:serverlessrepo:us-west-2:233765244907:applications/EdgeDelta-Forwarder-AMD64","serverlessrepo:semanticVersion":"0.0.6","tag1":"value1"}},"aws":{"log.group.name":"/aws/lambda/example-lambda","log.group.arn":"arn:aws:logs:us-east-2:000000000000:log-group:/aws/lambda/example-lambda","log.group.tags":{"tag2":"value2"},"log.stream.name":"2024/11/19/[$LATEST]e3199c8fcb374f978e12a020acc78362","log.message_type":"DATA_MESSAGE","log.subscription_filters":["cw1"]},"host.arch":"x86_64","process.runtime.name":"python3.13","logEvents":[{"id":"38626090287414021239190390440846764370530496153856770048","timestamp":1732053792084,"message":"INIT_START Runtime Version: python:3.13.v13\tRuntime Version ARN: arn:aws:lambda:us-east-2::runtime:b881cbc9a10a8bcb3def9d9e9fe38f922bb36510a1d92d4ce85cf2a899eeabd8\n"},{"id":"38626090289554892778249330262434193324704738858430889985","timestamp":1732053792180,"message":"START RequestId: 0cfd4e3c-cc4c-4bd7-8cb1-e2901fbb629b Version: $LATEST\n"},{"id":"38626090289577193523447860885575729042977387219936870402","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ\n"},{"id":"38626090289577193523447860885575729042977387219936870403","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181543315\n"},{"id":"38626090289577193523447860885575729042977387219936870404","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181554207\n"},{"id":"38626090289577193523447860885575729042977387219936870405","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181562744\n"},{"id":"38626090289577193523447860885575729042977387219936870406","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181568639\n"},{"id":"38626090289577193523447860885575729042977387219936870407","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181574403\n"},{"id":"38626090289577193523447860885575729042977387219936870408","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181581023\n"},{"id":"38626090289577193523447860885575729042977387219936870409","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181587230\n"},{"id":"38626090289577193523447860885575729042977387219936870410","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181592960\n"},{"id":"38626090289577193523447860885575729042977387219936870411","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181598519\n"},{"id":"38626090289577193523447860885575729042977387219936870412","timestamp":1732053792181,"message":"example-lambda-iNhPkYbSKJ-1732053792181605151\n"},{"id":"38626090289644095759043452755000336197795332304454811661","timestamp":1732053792184,"message":"END RequestId: 0cfd4e3c-cc4c-4bd7-8cb1-e2901fbb629b\n"},{"id":"38626090289644095759043452755000336197795332304454811662","timestamp":1732053792184,"message":"REPORT RequestId: 0cfd4e3c-cc4c-4bd7-8cb1-e2901fbb629b\tDuration: 2.06 ms\tBilled Duration: 3 ms\tMemory Size: 128 MB\tMax Memory Used: 31 MB\tInit Duration: 93.73 ms\t\n"}]}