Edge Delta JSON Unroll Node
4 minute read
Overview
The JSON Unroll Node transforms structured JSON logs by unrolling nested JSON array objects into separate logs, making the data easier to monitor and analyze. Each output log inherits the top level fields from the source log.
Example Configuration - Unroll from Body
The following configuration unrolls data from the Records fields in each log, and creates a new log for each record. Each generated log consists of a Record field at the top level within the body.
nodes:
- name: json_unroll
type: json_unroll
field_path: Records
new_field_name: Record
Example Input
Note: Sensitive information has been replaced with dummy data.
{
"Records": [
"0": {
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"invokedBy": "securityhub.amazonaws.com"
},
"eventTime": "2024-07-17T09:48:41Z",
"eventSource": "config.amazonaws.com",
"eventName": "DescribeEventAggregates",
"awsRegion": "us-west-2",
"sourceIPAddress": "13.71.17.166",
"userAgent": "config.amazonaws.com",
"requestParameters": {
"roleArn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789",
"roleSessionName": "AWSConfig-BucketConfigCheck"
},
"responseElements": {
"credentials": {
"accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
"expiration": "Jul 10, 171717 8:10:24 AM",
"sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLM"
},
"assumedRoleUser": {
"assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
"arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"
}
},
"requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
"eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
"readOnly": "true",
"resources": [
"0": {
"accountId": "123456789012",
"type": "AWS::IAM::Role",
"ARN": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789"
}
"length": 1,
],
"eventType": "AwsApiCall",
"managementEvent": "true",
"recipientAccountId": "123456789012",
"sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
"eventCategory": "Management"
},
"1": {
"eventVersion": "1.08",
"userIdentity": {
"type": "SAMLUser",
"invokedBy": "config.amazonaws.com"
},
"eventTime": "2024-07-17T09:48:41Z",
"eventSource": "ec2.amazonaws.com",
"eventName": "GetBucketAcl",
"awsRegion": "us-west-2",
"sourceIPAddress": "78.28.127.254",
"userAgent": "config.amazonaws.com",
"requestParameters": {
"roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
"roleSessionName": "AWSConfig"
},
"responseElements": {
"credentials": {
"accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
"expiration": "Jul 10, 171717 8:10:24 AM",
"sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGXYZ12"
},
"assumedRoleUser": {
"assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig",
"arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig"
}
},
"requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
"eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
"readOnly": "true",
"resources": [
"0": {
"accountId": "098765432109",
"type": "AWS::IAM::Role",
"ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210"
}
"length": 1,
],
"eventType": "AwsApiCall",
"managementEvent": "true",
"recipientAccountId": "098765432109",
"sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
"eventCategory": "Management"
},
...
Example Output
After being unrolled, the source log is split into individual logs - one per record. This results in more logs but each log is shorter. Here is the unrolled log for Record 1:
{
"Record": {
"awsRegion": "us-west-2",
"eventCategory": "Management",
"eventID": "qrst6789-efgh-1234-abcd-6789mnopuvwx",
"eventName": "DescribeEventAggregates",
"eventSource": "ec2.amazonaws.com",
"eventTime": "2024-07-17T09:48:41Z",
"eventType": "AwsApiCall",
"eventVersion": "1.08",
"managementEvent": "true",
"readOnly": "true",
"recipientAccountId": "098765432109",
"requestID": "wxyz9876-vuts-5432-rqpo-8765nmlkjihgfedc",
"requestParameters": {
"roleArn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
"roleSessionName": "AAWSConfig"
},
"resources": [
{
"ARN": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210",
"accountId": "098765432109",
"type": "AWS::IAM::Role"
}
],
"responseElements": {
"assumedRoleUser": {
"arn": "arn:aws:iam::987654321098:role/ZYXWVUTSRQPONML9876543210/AWSConfig-BucketConfigCheck",
"assumedRoleId": "B2C3D4E5F6G7H8I9J0A1:AWSConfig"
},
"credentials": {
"accessKeyId": "B2C3D4E5F6G7H8I9J0A1",
"expiration": "Jul 10, 171717 8:10:24 AM",
"sessionToken": "ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGHJKLMNPQRSTUVWXYZ23456789ABCDEFGXYZ12"
}
},
"sharedEventID": "01234567-89ab-bcde-dcba-9876543210fe",
"sourceIPAddress": "78.28.127.254",
"userAgent": "config.amazonaws.com",
"userIdentity": {
"invokedBy": "config.amazonaws.com",
"type": "SAMLUser"
}
}
}
Example Configuration 2 - Unroll from a Field
In this example, field_path specifies where the JSON data to be unrolled is located, json_field_path identifies the location of the array within the JSON to be unrolled.
- name: json_unroller
type: json_unroll
field_path: item["attributes"]["newBody"]
json_field_path: content["records"]
Example Input
{
"attributes": {
"newBody": {
"content": {
"records": [
{
"id": "1",
"message": "This is the first message."
},
{
"id": "2",
"message": "This is the second message."
}
]
}
}
}
}
Example Output
{
"record": {
"id": "1",
"message": "This is the first message."
}
}
{
"record": {
"id": "2",
"message": "This is the second message."
}
}
This example unrolls each entry within the content[“records”] array into individual logs, making the data easier to handle and analyze.
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: json_unroll
The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
field_path
The field_path parameter defines the location of the array that should be unrolled into individual logs. It is written as a string, but dot notation with array indexing is supported. A path field is required.
In this example the field path will access the metrics array of the first alarm.
nodes:
- name: json_unroll
type: json_unroll
field_path: cloudwatch.alarms.[0].metrics
Optional Parameters
json_field_path
The json_field_path parameter specifies the location in the json object of the array that needs to be unrolled. It is specified as a string and is optional.
- name: json_unroller
type: json_unroll
field_path: item["attributes"]["newBody"]
json_field_path: content["records"]
new_field_name
The new_field_name parameter specifies the field under which to place the unrolled log contents within the body. It is specified as a string and is optional. If it is left out, the original array’s name will be used.
nodes:
- name: json_unroll
type: json_unroll
field_path: cloudwatch.alarms.[0].metrics
new_field_name: metric