Edge Delta JSON Unroll Node

Unroll structured JSON content.

  4 minute read  

Overview

The JSON Unroll Node transforms structured JSON logs by unrolling nested JSON array objects into separate logs, making the data easier to monitor and analyze. Each output log inherits the top level fields from the source log.

  • incoming_data_types: log
  • outgoing_data_types: log

Example Configuration - Unroll from Body

The following configuration unrolls data from the Records fields in each log, and creates a new log for each record. Each generated log consists of a Record field at the top level within the body.

nodes:
- name: json_unroll
  type: json_unroll
  json_field_path: Records
  new_field_name: Record

Example Input

Note: Sensitive information has been replaced with dummy data.

{
  "Records": [
    {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "AssumedRole",
        "invokedBy": "securityhub.amazonaws.com"
      },
      "eventTime": "2024-07-17T09:48:41Z",
      "eventSource": "config.amazonaws.com",
      "eventName": "DescribeEventAggregates",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "13.71.17.166",
      "userAgent": "config.amazonaws.com",
      "requestParameters": {
        "roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
        "roleSessionName": "AWSConfig-BucketConfigCheck"
      },
      "responseElements": {
        "credentials": {
          "accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
          "expiration": "Jul 10, 171717 8:10:24 AM",
          "sessionToken": "..."
        },
        "assumedRoleUser": {
          "assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
          "arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"
        }
      },
      "requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
      "eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
      "readOnly": "true",
      "resources": [
        {
          "accountId": "123456789012",
          "type": "AWS::IAM::Role",
          "ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789"
        }
      ],
      "eventType": "AwsApiCall",
      "managementEvent": "true",
      "recipientAccountId": "123456789012",
      "sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
      "eventCategory": "Management"
    },
    {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "SAMLUser",
        "invokedBy": "config.amazonaws.com"
      },
      "eventTime": "2024-07-17T09:48:41Z",
      "eventSource": "ec2.amazonaws.com",
      "eventName": "GetBucketAcl",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "78.28.127.254",
      "userAgent": "config.amazonaws.com",
      "requestParameters": {
        "roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
        "roleSessionName": "AWSConfig"
      },
      "responseElements": {
        "credentials": {
          "accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
          "expiration": "Jul 10, 171717 8:10:24 AM",
          "sessionToken": "..."
        },
        "assumedRoleUser": {
          "assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig",
          "arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig"
        }
      },
      "requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
      "eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
      "readOnly": "true",
      "resources": [
        {
          "accountId": "098765432109",
          "type": "AWS::IAM::Role",
          "ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210"
        }
      ],
      "eventType": "AwsApiCall",
      "managementEvent": "true",
      "recipientAccountId": "098765432109",
      "sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
      "eventCategory": "Management"
    }
  ]
}

For convenience, here is the log as a single line:

{"Records":[{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","invokedBy":"securityhub.amazonaws.com"},"eventTime":"2024-07-17T09:48:41Z","eventSource":"config.amazonaws.com","eventName":"DescribeEventAggregates","awsRegion":"us-west-2","sourceIPAddress":"13.71.17.166","userAgent":"config.amazonaws.com","requestParameters":{"roleArn":"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789","roleSessionName":"AWSConfig-BucketConfigCheck"},"responseElements":{"credentials":{"accessKeyId":"A1B2C3D4E5F6G7H8I9J0","expiration":"Jul 10, 171717 8:10:24 AM","sessionToken":"..."},"assumedRoleUser":{"assumedRoleId":"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck","arn":"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"}},"requestID":"abcd1234-efgh-5678-ijkl-9012mnopqrst","eventID":"mnop5678-abcd-1234-efgh-5678ijklqrst","readOnly":"true","resources":[{"accountId":"123456789012","type":"AWS::IAM::Role","ARN":"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789"}],"eventType":"AwsApiCall","managementEvent":"true","recipientAccountId":"123456789012","sharedEventID":"01234567-89ab-cdef-edcb-a9876543210f","eventCategory":"Management"},{"eventVersion":"1.08","userIdentity":{"type":"SAMLUser","invokedBy":"config.amazonaws.com"},"eventTime":"2024-07-17T09:48:41Z","eventSource":"ec2.amazonaws.com","eventName":"GetBucketAcl","awsRegion":"us-west-2","sourceIPAddress":"78.28.127.254","userAgent":"config.amazonaws.com","requestParameters":{"roleArn":"arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210","roleSessionName":"AWSConfig"},"responseElements":{"credentials":{"accessKeyId":"B2C3D4E5F6G7H8I9J0A1","expiration":"Jul 10, 171717 8:10:24 AM","sessionToken":"..."},"assumedRoleUser":{"assumedRoleId":"B2C3D4E5F6G7H8I9J0A1:AWSConfig","arn":"arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig"}},"requestID":"wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc","eventID":"qrst6789-efgh-1234-abcd-6789mnopuvwx","readOnly":"true","resources":[{"accountId":"098765432109","type":"AWS::IAM::Role","ARN":"arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210"}],"eventType":"AwsApiCall","managementEvent":"true","recipientAccountId":"098765432109","sharedEventID":"01234567-89ab-bcde-dcba-9876543210fe","eventCategory":"Management"}]}

Example Output

After being unrolled, the source log is split into individual logs - one per record. This results in more logs but each log is shorter. Here is the unrolled log for the first Record:

{
  "Record": {
    "awsRegion": "us-west-2",
    "eventCategory": "Management",
    "eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
    "eventName": "DescribeEventAggregates",
    "eventSource": "config.amazonaws.com",
    "eventTime": "2024-07-17T09:48:41Z",
    "eventType": "AwsApiCall",
    "eventVersion": "1.08",
    "managementEvent": "true",
    "readOnly": "true",
    "recipientAccountId": "123456789012",
    "requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
    "requestParameters": {
      "roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
      "roleSessionName": "AWSConfig-BucketConfigCheck"
    },
    "resources": [
      "0": {
        "ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
        "accountId": "123456789012",
        "type": "AWS::IAM::Role"
      }
      "length": 1,
    ],
    "responseElements": {
      "assumedRoleUser": {
        "arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck",
        "assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck"
      },
      "credentials": {
        "accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
        "expiration": "Jul 10, 171717 8:10:24 AM",
        "sessionToken": "..."
      }
    },
    "sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
    "sourceIPAddress": "13.71.17.166",
    "userAgent": "config.amazonaws.com",
    "userIdentity": {
      "invokedBy": "securityhub.amazonaws.com",
      "type": "AssumedRole"
    }
  }
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: json_unroll

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

field_path

The field_path parameter defines the location of the array that should be unrolled into individual logs. It is written as a string, but dot notation with array indexing is supported. A path field is required.

In this example the field path will access the metrics array of the first alarm.

nodes:
- name: json_unroll
  type: json_unroll
  field_path: cloudwatch.alarms.[0].metrics

Optional Parameters

json_field_path

The json_field_path parameter specifies the location in the JSON object of the array that needs to be unrolled. It is specified as a string and is optional.

- name: json_unroller
  type: json_unroll
  field_path: item["attributes"]["newBody"]
  json_field_path: content["records"]

new_field_name

The new_field_name parameter specifies the field under which to place the unrolled log contents within the body. It is specified as a string and is optional. If it is left out, the original array’s name will be used.

nodes:
- name: json_unroll
  type: json_unroll
  field_path: cloudwatch.alarms.[0].metrics
  new_field_name: metric