keep_matching_keys

Learn about the keep_matching_keys OTTL editor function.

  2 minute read  

This function is used to retain only the keys from a specified target that match a given pattern. It helps streamline data by keeping desired entries and removing non-matching ones.

Syntax: keep_matching_keys(target, pattern)

  • Target: The target refers to the field or object within the log entry where you want to retain keys. It usually specifies a parent container, like a JSON object or associative array, which holds multiple key-value pairs.
  • Pattern: The pattern is a regular expression designating which keys to keep within the target. Only keys that match this pattern will be retained along with their associated values.

Input

{
	"_type": "log",
	"attributes": {
		"parsed_body": {
			"awsRegion": "us-west-2",
			"eventCategory": "Management",
			"eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
			"eventName": "ListStacks",
			"eventSource": "ec2.amazonaws.com",
			"eventTime": "2024-12-05T05:10:57.227003Z",
			"eventType": "AwsApiCall",
			"eventVersion": "1.08",
			"managementEvent": "true",
			"readOnly": "true",
			"recipientAccountId": 123456789012,
			"requestID": "abcd1234-efgh-5678-ijkl-9012mnopqrst",
			"resources.0": {
				"ARN": "arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole",
				"accountId": 123456789012,
				"type": "AWS::IAM::Role"
			},
			"responseElements.assumedRoleUser.arn": "arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck",
			"responseElements.assumedRoleUser.assumedRoleId": "A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck",
			"responseElements.credentials.accessKeyId": "A1B2C3D4E5F6G7H8I9J0",
			"responseElements.credentials.expiration": "2024-12-05T05:10:57.227053Z",
			"responseElements.credentials.sessionToken": "123456789876",
			"sharedEventID": "01234567-89ab-cdef-edcb-a9876543210f",
			"sourceIPAddress": "211.46.216.146",
			"userAgent": "ec2.amazonaws.com",
			"userIdentity.invokedBy": "lambda.amazonaws.com",
			"userIdentity.type": "AssumedRole"
		}
	},
	"body": "{\"eventVersion\": \"1.08\", \"userIdentity\": {\"type\": \"AssumedRole\", \"invokedBy\": \"lambda.amazonaws.com\"}, \"eventTime\": \"2024-12-05T05:10:57.227003Z\", \"eventSource\": \"ec2.amazonaws.com\", \"eventName\": \"ListStacks\", \"awsRegion\": \"us-west-2\", \"sourceIPAddress\": \"211.46.216.146\", \"userAgent\": \"ec2.amazonaws.com\", \"requestParameters\": {}, \"responseElements\": {\"credentials\": {\"accessKeyId\": \"A1B2C3D4E5F6G7H8I9J0\", \"expiration\": \"2024-12-05T05:10:57.227053Z\", \"sessionToken\": \"123456789876\"}, \"assumedRoleUser\": {\"assumedRoleId\": \"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck\", \"arn\": \"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck\"}}, \"requestID\": \"abcd1234-efgh-5678-ijkl-9012mnopqrst\", \"eventID\": \"mnop5678-abcd-1234-efgh-5678ijklqrst\", \"readOnly\": \"true\", \"resources\": [{\"accountId\": 123456789012, \"type\": \"AWS::IAM::Role\", \"ARN\": \"arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole\"}], \"eventType\": \"AwsApiCall\", \"managementEvent\": \"true\", \"recipientAccountId\": 123456789012, \"sharedEventID\": \"01234567-89ab-cdef-edcb-a9876543210f\", \"eventCategory\": \"Management\"}",
	"resource": {...},
	"timestamp": 1733378455697
}

Statement

keep_matching_keys(attributes["parsed_body"], "^event.*")

Output

{
	"_type": "log",
	"attributes": {
		"parsed_body": {
			"eventCategory": "Management",
			"eventID": "mnop5678-abcd-1234-efgh-5678ijklqrst",
			"eventName": "ListStacks",
			"eventSource": "ec2.amazonaws.com",
			"eventTime": "2024-12-05T05:10:57.227003Z",
			"eventType": "AwsApiCall",
			"eventVersion": "1.08"
		}
	},
	"body": "{\"eventVersion\": \"1.08\", \"userIdentity\": {\"type\": \"AssumedRole\", \"invokedBy\": \"lambda.amazonaws.com\"}, \"eventTime\": \"2024-12-05T05:10:57.227003Z\", \"eventSource\": \"ec2.amazonaws.com\", \"eventName\": \"ListStacks\", \"awsRegion\": \"us-west-2\", \"sourceIPAddress\": \"211.46.216.146\", \"userAgent\": \"ec2.amazonaws.com\", \"requestParameters\": {}, \"responseElements\": {\"credentials\": {\"accessKeyId\": \"A1B2C3D4E5F6G7H8I9J0\", \"expiration\": \"2024-12-05T05:10:57.227053Z\", \"sessionToken\": \"123456789876\"}, \"assumedRoleUser\": {\"assumedRoleId\": \"A1B2C3D4E5F6G7H8I9J0:AWSConfig-BucketConfigCheck\", \"arn\": \"arn:aws:iam::123456789012:role/ABCDEFGHIJKLM123456789/AWSConfig-BucketConfigCheck\"}}, \"requestID\": \"abcd1234-efgh-5678-ijkl-9012mnopqrst\", \"eventID\": \"mnop5678-abcd-1234-efgh-5678ijklqrst\", \"readOnly\": \"true\", \"resources\": [{\"accountId\": 123456789012, \"type\": \"AWS::IAM::Role\", \"ARN\": \"arn:aws:iam::123456789012:role/aws-controltower-ForwardSnsNotificationRole\"}], \"eventType\": \"AwsApiCall\", \"managementEvent\": \"true\", \"recipientAccountId\": 123456789012, \"sharedEventID\": \"01234567-89ab-cdef-edcb-a9876543210f\", \"eventCategory\": \"Management\"}",
	"resource": {...},
	"timestamp": 1733378478884
}

The log entry retains only the keys within the attributes["parsed_body"] object that match the pattern ^event.*, for example, keeping the eventCategory and eventID keys, while removing all others.