Edge Delta Unroll JSON Processor

The Edge Delta Unroll JSON processor unrolls events from a single JSON object.

5 minute read

Overview

The JSON Unroll processor transforms structured JSON logs by unrolling nested JSON array objects into separate logs, making the data easier to monitor and analyze. Each output log inherits the top level fields and resources from the source log.

incoming_data_types: log
outgoing_data_types: log

Example Configuration - Unroll from Body

The following configuration unrolls data from the Records fields in each log, and creates a new log for each record. Each generated log consists of a Record field at the top level within the body.

In this example the single input log has been unrolled into two distinct logs - one per event.

nodes:
- name: kubernetes_input_ab15_multiprocessor
  type: sequence
  user_description: Multi Processor
  processors:
  - type: json_unroll
    metadata: '{"id":"o8wIHKtPQX8wBgJ3EUUoO","type":"json_unroll","name":"JSON Unroll"}'
    data_types:
    - log
    field_path: body
    new_field_name: Record
    json_field_path: Records

Example Input

Note: Sensitive information has been replaced with dummy data.

{
  "Records": [
    {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "AssumedRole",
        "invokedBy": "secondexample.amazonaws.com"
      },
      "eventTime": "2024-07-17T09:48:41Z",
      "eventSource": "config.amazonaws.com",
      "eventName": "DescribeEventAggregates",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "13.71.17.166",
      "userAgent": "config.amazonaws.com",
      "requestParameters": {
        "roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
        "roleSessionName": "AWSConfig-BucketConfigCheck"
      },
      "responseElements": {
        "credentials": {
          "accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
          "expiration": "2024-07-17T10:10:24Z",
          "sessionToken": "token123456"
        },
        "assumedRoleUser": {
          "assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
          "arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"
        }
      },
      "requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
      "eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
      "readOnly": true,
      "resources": [
        {
          "accountId": "123456789012",
          "type": "AWS::IAM::Role",
          "ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789"
        }
      ],
      "eventType": "AwsApiCall",
      "managementEvent": true,
      "recipientAccountId": "123456789012",
      "sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
      "eventCategory": "Management"
    },
    {
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "SAMLUser",
        "invokedBy": "config.amazonaws.com"
      },
      "eventTime": "2024-07-17T09:48:41Z",
      "eventSource": "ec2.amazonaws.com",
      "eventName": "GetBucketAcl",
      "awsRegion": "us-west-2",
      "sourceIPAddress": "78.28.127.254",
      "userAgent": "config.amazonaws.com",
      "requestParameters": {
        "roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
        "roleSessionName": "AWSConfig"
      },
      "responseElements": {
        "credentials": {
          "accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
          "expiration": "2024-07-17T10:10:24Z",
          "sessionToken": "token654321"
        },
        "assumedRoleUser": {
          "assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig",
          "arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig"
        }
      },
      "requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
      "eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
      "readOnly": true,
      "resources": [
        {
          "accountId": "098765432109",
          "type": "AWS::IAM::Role",
          "ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210"
        }
      ],
      "eventType": "AwsApiCall",
      "managementEvent": true,
      "recipientAccountId": "098765432109",
      "sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
      "eventCategory": "Management3"
    }
  ]
}

Example Output

After being unrolled, the source log is split into individual logs - one per record. This results in more logs but each log is shorter. Here is the unrolled log for the first Record:

Log 1:

{
  "Record": {
    "awsRegion": "us-west-2",
    "eventCategory": "Management",
    "eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
    "eventName": "DescribeEventAggregates",
    "eventSource": "config.amazonaws.com",
    "eventTime": "2024-07-17T09:48:41Z",
    "eventType": "AwsApiCall",
    "eventVersion": "1.08",
    "managementEvent": true,
    "readOnly": true,
    "recipientAccountId": "123456789012",
    "requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
    "requestParameters": {
      "roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
      "roleSessionName": "AWSConfig-BucketConfigCheck"
    },
    "resources": [
      {
        "ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
        "accountId": "123456789012",
        "type": "AWS::IAM::Role"
      }
    ],
    "responseElements": {
      "assumedRoleUser": {
        "arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck",
        "assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck"
      },
      "credentials": {
        "accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
        "expiration": "2024-07-17T10:10:24Z",
        "sessionToken": "token123456"
      }
    },
    "sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
    "sourceIPAddress": "13.71.17.166",
    "userAgent": "config.amazonaws.com",
    "userIdentity": {
      "invokedBy": "secondexample.amazonaws.com",
      "type": "AssumedRole"
    }
  }
}

Log 2:

{
  "Record": {
    "awsRegion": "us-west-2",
    "eventCategory": "Management3",
    "eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
    "eventName": "GetBucketAcl",
    "eventSource": "ec2.amazonaws.com",
    "eventTime": "2024-07-17T09:48:41Z",
    "eventType": "AwsApiCall",
    "eventVersion": "1.08",
    "managementEvent": true,
    "readOnly": true,
    "recipientAccountId": "098765432109",
    "requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
    "requestParameters": {
      "roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
      "roleSessionName": "AWSConfig"
    },
    "resources": [
      {
        "ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
        "accountId": "098765432109",
        "type": "AWS::IAM::Role"
      }
    ],
    "responseElements": {
      "assumedRoleUser": {
        "arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig",
        "assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig"
      },
      "credentials": {
        "accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
        "expiration": "2024-07-17T10:10:24Z",
        "sessionToken": "token654321"
      }
    },
    "sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
    "sourceIPAddress": "78.28.127.254",
    "userAgent": "config.amazonaws.com",
    "userIdentity": {
      "invokedBy": "config.amazonaws.com",
      "type": "SAMLUser"
    }
  }
}

Options

condition

The condition parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional. You can select one of the following operators:

Operator	Name	Description	Example
`==`	Equal to	Returns `true` if both values are exactly the same	`attributes["status"] == "OK"`
`!=`	Not equal to	Returns `true` if the values are not the same	`attributes["level"] != "debug"`
`>`	Greater than	Returns `true` if the left value is greater than the right	`attributes["duration_ms"] > 1000`
`>=`	Greater than or equal	Returns `true` if the left value is greater than or equal to the right	`attributes["score"] >= 90`
`<`	Less than	Returns `true` if the left value is less than the right	`attributes["load"] < 0.75`
`<=`	Less than or equal	Returns `true` if the left value is less than or equal to the right	`attributes["retries"] <= 3`
`matches`	Regex match	Returns `true` if the string matches a regular expression	`isMatch(attributes["name"], ".*\\.name$"`

It is defined in YAML as follows:

- name: _multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    condition: attributes["request"]["path"] == "/json/view"

Parse from

This option specifies the field containing the JSON object that needs to be unrolled. It is specified using bracket notation and is optional. If left empty it defaults to body.

JSON Field path

This option specifies the location in the JSON object of the array that needs to be unrolled. It is specified as a string and is optional.

New field name

This parameter specifies the field under which to place the unrolled log contents within the body. It is specified as a string and is optional. If it is left out, the original array’s name will be used.

Final

The final parameter specifies whether successfully processed data items should continue to subsequent processors within the same multiprocessor node. Data items that fail to be processed by the processor will be passed to the next processor in the node regardless of this setting. You select the slider in the tool which specifies it for you in the YAML as a Boolean. The default is false and it is optional.

It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
    - type: <processor type>
    final: true

Keep original telemetry item

This option defines whether to delete the original unmodified data item after it is processed. For example, you can keep the original log as well as any metrics generated by an extract metric processor. If you select this option your data volume will increase.