flatten

Learn about the flatten OTTL editor function.

  4 minute read  

This function is used to convert nested structures within a log entry into a flat format, typically by turning nested paths into a single level of key-value pairs. It’s particularly useful for simplifying data access and storage when dealing with complex nested structures.

Syntax: flatten(target)

  • Target: The target is the field or object within the log entry that you wish to flatten. It often involves nested JSON objects or arrays which are to be transformed into a simpler structure.

Imagine the agent ingests this log message:

{"eventVersion": "1.08", "userIdentity": {"type": "AssumedRole", "invokedBy": "lambda.amazonaws.com"}, "eventTime": "2024-12-05T05:10:57.227003Z", "eventSource": "ec2.amazonaws.com", "eventName": "ListStacks", "awsRegion": "us-west-2", "sourceIPAddress": "211.46.216.146", "userAgent": "ec2.amazonaws.com", "requestParameters": {}, "responseElements": {"credentials": {"accessKeyId": "A1B2C3D4E5F6G7H8I9J0", "expiration": "2024-12-05T05:10:57.227053Z", "sessionToken": "123456789876"}, "assumedRoleUser": {"assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck", "arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck"}}, "requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst", "eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst", "readOnly": "true", "resources": [{"accountId": 123456789012, "type": "AWS::IAM::Role", "ARN": "arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole"}], "eventType": "AwsApiCall", "managementEvent": "true", "recipientAccountId": 123456789012, "sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f", "eventCategory": "Management"}

Bear in mind the log is escaped when ingested. See Understand Escaping Characters.

In this example, assume the following OTTL statements have been executed on the log:

set(attributes["decoded_body"], Decode(body, "utf-8"))
set(attributes["parsed_body"], ParseJSON(attributes["decoded_body"]))

To start, the body was decoded from a byte array. See Working with the body for more information about decoding the body.

Next the JSON object was parsed into nested key value pairs. Now the log is ready to be flattened.

Input

{
	"_type": "log",
	"attributes": {
		"decoded_body": "{\"eventVersion\": \"1.08\", \"userIdentity\": {\"type\": \"AssumedRole\", \"invokedBy\": \"lambda.amazonaws.com\"}, \"eventTime\": \"2024-12-05T05:10:57.227003Z\", \"eventSource\": \"ec2.amazonaws.com\", \"eventName\": \"ListStacks\", \"awsRegion\": \"us-west-2\", \"sourceIPAddress\": \"211.46.216.146\", \"userAgent\": \"ec2.amazonaws.com\", \"requestParameters\": {}, \"responseElements\": {\"credentials\": {\"accessKeyId\": \"A1B2C3D4E5F6G7H8I9J0\", \"expiration\": \"2024-12-05T05:10:57.227053Z\", \"sessionToken\": \"123456789876\"}, \"assumedRoleUser\": {\"assumedRoleId\": \"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck\", \"arn\": \"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck\"}}, \"requestID\": \"abcd1234-efgh-5678-ijkl-9012mnopqrst\", \"eventID\": \"mnop5678-abcd-1234-efgh-5678ijklqrst\", \"readOnly\": \"true\", \"resources\": [{\"accountId\": 123456789012, \"type\": \"AWS::IAM::Role\", \"ARN\": \"arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole\"}], \"eventType\": \"AwsApiCall\", \"managementEvent\": \"true\", \"recipientAccountId\": 123456789012, \"sharedEventID\": \"01234567-89ab-cdef-edcb-a9876543210f\", \"eventCategory\": \"Management\"}",
		"parsed_body": {
			"awsRegion": "us-west-2",
			"eventCategory": "Management",
			"eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
			"eventName": "ListStacks",
			"eventSource": "ec2.amazonaws.com",
			"eventTime": "2024-12-05T05:10:57.227003Z",
			"eventType": "AwsApiCall",
			"eventVersion": "1.08",
			"managementEvent": "true",
			"readOnly": "true",
			"recipientAccountId": 123456789012,
			"requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
			"requestParameters": {},
			"resources": [
				{
					"ARN": "arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole",
					"accountId": 123456789012,
					"type": "AWS::IAM::Role"
				}
			],
			"responseElements": {
				"assumedRoleUser": {
					"arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck",
					"assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck"
				},
				"credentials": {
					"accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
					"expiration": "2024-12-05T05:10:57.227053Z",
					"sessionToken": "123456789876"
				}
			},
			"sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
			"sourceIPAddress": "211.46.216.146",
			"userAgent": "ec2.amazonaws.com",
			"userIdentity": {
				"invokedBy": "lambda.amazonaws.com",
				"type": "AssumedRole"
			}
		}
	},
	"body": "{\"eventVersion\": \"1.08\", \"userIdentity\": {\"type\": \"AssumedRole\", \"invokedBy\": \"lambda.amazonaws.com\"}, \"eventTime\": \"2024-12-05T05:10:57.227003Z\", \"eventSource\": \"ec2.amazonaws.com\", \"eventName\": \"ListStacks\", \"awsRegion\": \"us-west-2\", \"sourceIPAddress\": \"211.46.216.146\", \"userAgent\": \"ec2.amazonaws.com\", \"requestParameters\": {}, \"responseElements\": {\"credentials\": {\"accessKeyId\": \"A1B2C3D4E5F6G7H8I9J0\", \"expiration\": \"2024-12-05T05:10:57.227053Z\", \"sessionToken\": \"123456789876\"}, \"assumedRoleUser\": {\"assumedRoleId\": \"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck\", \"arn\": \"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck\"}}, \"requestID\": \"abcd1234-efgh-5678-ijkl-9012mnopqrst\", \"eventID\": \"mnop5678-abcd-1234-efgh-5678ijklqrst\", \"readOnly\": \"true\", \"resources\": [{\"accountId\": 123456789012, \"type\": \"AWS::IAM::Role\", \"ARN\": \"arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole\"}], \"eventType\": \"AwsApiCall\", \"managementEvent\": \"true\", \"recipientAccountId\": 123456789012, \"sharedEventID\": \"01234567-89ab-cdef-edcb-a9876543210f\", \"eventCategory\": \"Management\"}",
	"resource": {...},
	"timestamp": 1733376154621
}

Statement

flatten(attributes["parsed_body"])

Output

{
	"_type": "log",
	"attributes": {
		"decoded_body": "{\"eventVersion\": \"1.08\", \"userIdentity\": {\"type\": \"AssumedRole\", \"invokedBy\": \"lambda.amazonaws.com\"}, \"eventTime\": \"2024-12-05T05:10:57.227003Z\", \"eventSource\": \"ec2.amazonaws.com\", \"eventName\": \"ListStacks\", \"awsRegion\": \"us-west-2\", \"sourceIPAddress\": \"211.46.216.146\", \"userAgent\": \"ec2.amazonaws.com\", \"requestParameters\": {}, \"responseElements\": {\"credentials\": {\"accessKeyId\": \"A1B2C3D4E5F6G7H8I9J0\", \"expiration\": \"2024-12-05T05:10:57.227053Z\", \"sessionToken\": \"123456789876\"}, \"assumedRoleUser\": {\"assumedRoleId\": \"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck\", \"arn\": \"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck\"}}, \"requestID\": \"abcd1234-efgh-5678-ijkl-9012mnopqrst\", \"eventID\": \"mnop5678-abcd-1234-efgh-5678ijklqrst\", \"readOnly\": \"true\", \"resources\": [{\"accountId\": 123456789012, \"type\": \"AWS::IAM::Role\", \"ARN\": \"arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole\"}], \"eventType\": \"AwsApiCall\", \"managementEvent\": \"true\", \"recipientAccountId\": 123456789012, \"sharedEventID\": \"01234567-89ab-cdef-edcb-a9876543210f\", \"eventCategory\": \"Management\"}",
		"parsed_body": {
			"awsRegion": "us-west-2",
			"eventCategory": "Management",
			"eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
			"eventName": "ListStacks",
			"eventSource": "ec2.amazonaws.com",
			"eventTime": "2024-12-05T05:10:57.227003Z",
			"eventType": "AwsApiCall",
			"eventVersion": "1.08",
			"managementEvent": "true",
			"readOnly": "true",
			"recipientAccountId": 123456789012,
			"requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
			"resources.0": {
				"ARN": "arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole",
				"accountId": 123456789012,
				"type": "AWS::IAM::Role"
			},
			"responseElements.assumedRoleUser.arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck",
			"responseElements.assumedRoleUser.assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
			"responseElements.credentials.accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
			"responseElements.credentials.expiration": "2024-12-05T05:10:57.227053Z",
			"responseElements.credentials.sessionToken": "123456789876",
			"sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
			"sourceIPAddress": "211.46.216.146",
			"userAgent": "ec2.amazonaws.com",
			"userIdentity.invokedBy": "lambda.amazonaws.com",
			"userIdentity.type": "AssumedRole"
		}
	},
	"body": "{\"eventVersion\": \"1.08\", \"userIdentity\": {\"type\": \"AssumedRole\", \"invokedBy\": \"lambda.amazonaws.com\"}, \"eventTime\": \"2024-12-05T05:10:57.227003Z\", \"eventSource\": \"ec2.amazonaws.com\", \"eventName\": \"ListStacks\", \"awsRegion\": \"us-west-2\", \"sourceIPAddress\": \"211.46.216.146\", \"userAgent\": \"ec2.amazonaws.com\", \"requestParameters\": {}, \"responseElements\": {\"credentials\": {\"accessKeyId\": \"A1B2C3D4E5F6G7H8I9J0\", \"expiration\": \"2024-12-05T05:10:57.227053Z\", \"sessionToken\": \"123456789876\"}, \"assumedRoleUser\": {\"assumedRoleId\": \"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck\", \"arn\": \"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck\"}}, \"requestID\": \"abcd1234-efgh-5678-ijkl-9012mnopqrst\", \"eventID\": \"mnop5678-abcd-1234-efgh-5678ijklqrst\", \"readOnly\": \"true\", \"resources\": [{\"accountId\": 123456789012, \"type\": \"AWS::IAM::Role\", \"ARN\": \"arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole\"}], \"eventType\": \"AwsApiCall\", \"managementEvent\": \"true\", \"recipientAccountId\": 123456789012, \"sharedEventID\": \"01234567-89ab-cdef-edcb-a9876543210f\", \"eventCategory\": \"Management\"}",
	"resource": {...},
	"timestamp": 1733376208046
}

In this example, the flatten function is used to transform the nested JSON structure in the parsed_body attribute of the log into a single level of key-value pairs. The output demonstrates how each nested level is converted to a flat format by appending the parent keys as prefixes to their child attributes. This transformation reduces the complexity of accessing deeply nested data. For instance, nested elements such as userIdentity which contains subfields like type and invokedBy, are transformed to userIdentity.type and userIdentity.invokedBy respectively in the flattened output.