Edge Delta's OTTL Extensions
20 minute read
Edge Delta Custom OTTL
Edge Delta has extended the OTTL with custom functionality. The extensions are organized into two categories: Converter Functions and Editor Functions.
Converter Functions
Converter functions transform, extract, or evaluate data and return values that can be assigned to fields.
EDXEncode
Minimum Agent Version: v1.25.0
EDXEncode fills a critical gap in standard OTTL by providing byte array encoding capabilities. While OTTL has a Decode function for converting byte arrays to strings, it lacks a corresponding encode function. This Edge Delta extension enables you to convert strings back to byte arrays, which is essential when working with the body field that must remain as a byte array.
When a log is ingested by the Edge Delta agent, a log data item is created and the log contents are placed in the body field. Importantly, this field is a byte array. Therefore, to perform any OTTL transformation that requires the body as a string input, it needs to be decoded from byte array to a string type as part of the transformation function. See Working with the body.
After copying, decoding and transforming the body into a field other than body, you can use the EDXEncode custom function to save a new body field in the appropriate byte array type.
Syntax
EDXEncode(target_field, current_format, target_encoding)
target_field: A field that should be converted to byte array.current_format: The current type from the IANA encoding index, such as “utf-8”; or “base64”.target_encoding: A boolean for the target type:truefor byte array,falsefor string.
Suppose this log is ingested:
session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData
The following operations are performed to extract, decode, parse, and transform the data:
set(attributes["decoded_body"], Decode(body, "utf-8"))
set(attributes["kv_map"], ParseKeyValue(attributes["decoded_body"]))
delete_matching_keys(attributes["kv_map"], "(temp_.*|debug_.*)")
set(attributes["new_body"], ToKeyValueString(attributes["kv_map"]))
See Decode, ParseKeyValue, delete_matching_keys, and ToKeyValueString.
These operations result in the following log:
Input:
{
"_type": "log",
"timestamp": 1735802408445,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"connection_id": "conn456",
"event_status": "success",
"event_type": "login",
"session_id": "abc123",
"user_id": "admin"
},
"new_body": "event_status=success connection_id=conn456 session_id=abc123 user_id=admin event_type=login"
}
}
Example:
set(body, EDXEncode(attributes["new_body"], "utf-8", true))
Output:
{
"_type": "log",
"timestamp": 1735802426292,
"body": "connection_id=conn456 session_id=abc123 user_id=admin event_type=login event_status=success",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"connection_id": "conn456",
"event_status": "success",
"event_type": "login",
"session_id": "abc123",
"user_id": "admin"
},
"new_body": "connection_id=conn456 session_id=abc123 user_id=admin event_type=login event_status=success"
}
}
The new_body attribute has been upserted into the body field and encoded as a byte array.
EDXCompress
Minimum Agent Version: v1.31.0
EDXCompress fills a gap in standard OTTL which lacks native compression capabilities. This Edge Delta extension enables you to compress strings or byte arrays using various algorithms (gzip, zlib, deflate, snappy, zstd), which is essential for reducing data size before storage or transmission in telemetry pipelines.
Syntax
EDXCompress(input, compression_type)
input: The string or byte array to compress.compression_type: The compression algorithm to use (e.g., “gzip”, “zlib”, “deflate”, “snappy”, “zstd”).
Input
{
"_type": "log",
"timestamp": 1735789200000,
"body": "This is a very long log message that contains repeated information. This is a very long log message that contains repeated information. This is a very long log message that contains repeated information.",
"resource": {...},
"attributes": {
"original_size": 198
},
"cache": {
"body": "This is a very long log message that contains repeated information. This is a very long log message that contains repeated information. This is a very long log message that contains repeated information."
}
}
Example
set(body, EDXCompress(cache["body"], "gzip"))
set(attributes["compression"], "gzip")
set(attributes["compressed_size"], Len(body))
Output
{
"_type": "log",
"timestamp": 1735789230000,
"body": "[compressed byte array]",
"resource": {...},
"attributes": {
"original_size": 198,
"compression": "gzip",
"compressed_size": 52
},
"cache": {
"body": "This is a very long log message that contains repeated information. This is a very long log message that contains repeated information. This is a very long log message that contains repeated information."
}
}
The body field now contains the compressed version of the original text as a byte array, significantly reducing the data size from 198 bytes to approximately 52 bytes when using gzip compression.
EDXDecompress
Minimum Agent Version: v1.31.0
EDXDecompress complements EDXCompress and fills a gap in standard OTTL which lacks native decompression capabilities. This Edge Delta extension enables you to decompress byte arrays that have been compressed with various algorithms, allowing you to work with compressed data sources in your telemetry pipeline.
Syntax
EDXDecompress(input, compression_type, return_as_string)
input: The compressed byte array to decompress.compression_type: The decompression algorithm to use (e.g., “gzip”, “zlib”, “deflate”, “snappy”, “zstd”).return_as_string: Boolean value -trueto return as string,falseto return as byte array.
Input
{
"_type": "log",
"timestamp": 1735789300000,
"body": "[compressed byte array]",
"resource": {...},
"attributes": {
"compression": "gzip",
"compressed_size": 52
},
"cache": {
"body": "[compressed byte array]"
}
}
Example
set(attributes["decompressed_body"], EDXDecompress(cache["body"], "gzip", true))
set(attributes["decompressed_size"], Len(attributes["decompressed_body"]))
Output
{
"_type": "log",
"timestamp": 1735789330000,
"body": "[compressed byte array]",
"resource": {...},
"attributes": {
"compression": "gzip",
"compressed_size": 52,
"decompressed_body": "This is a very long log message that contains repeated information. This is a very long log message that contains repeated information. This is a very long log message that contains repeated information.",
"decompressed_size": 198
},
"cache": {
"body": "[compressed byte array]"
}
}
The compressed data from cache["body"] has been decompressed and stored as a string in attributes["decompressed_body"], restoring the original 198-byte message from the 52-byte compressed format.
EDXUnescapeJSON
Minimum Agent Version: v1.31.0
EDXUnescapeJSON fills a gap in standard OTTL’s JSON handling capabilities. While OTTL provides ParseJSON for parsing JSON strings, it cannot handle multiply-escaped JSON strings that are common in nested logging scenarios. This Edge Delta extension recursively unescapes JSON strings, making it possible to properly parse deeply nested or repeatedly escaped JSON data.
Syntax
EDXUnescapeJSON(input)
input: The escaped JSON string to unescape.
Input
{
"_type": "log",
"timestamp": 1735789800000,
"body": "Processing escaped JSON data",
"resource": {...},
"attributes": {
"escaped_json": "{\\\"level\\\":\\\"info\\\",\\\"message\\\":\\\"User logged in\\\",\\\"metadata\\\":\\\"{\\\\\\\"user_id\\\\\\\":\\\\\\\"12345\\\\\\\",\\\\\\\"session\\\\\\\":\\\\\\\"abc-xyz\\\\\\\"}\\\"}"
},
"cache": {
"body": "{\\\"level\\\":\\\"info\\\",\\\"message\\\":\\\"User logged in\\\",\\\"metadata\\\":\\\"{\\\\\\\"user_id\\\\\\\":\\\\\\\"12345\\\\\\\",\\\\\\\"session\\\\\\\":\\\\\\\"abc-xyz\\\\\\\"}\\\"}"
}
}
Example
set(attributes["unescaped_json"], EDXUnescapeJSON(cache["body"]))
set(attributes["parsed_data"], ParseJSON(attributes["unescaped_json"]))
Output
{
"_type": "log",
"timestamp": 1735789830000,
"body": "Processing escaped JSON data",
"resource": {...},
"attributes": {
"escaped_json": "{\\\"level\\\":\\\"info\\\",\\\"message\\\":\\\"User logged in\\\",\\\"metadata\\\":\\\"{\\\\\\\"user_id\\\\\\\":\\\\\\\"12345\\\\\\\",\\\\\\\"session\\\\\\\":\\\\\\\"abc-xyz\\\\\\\"}\\\"}",
"unescaped_json": "{\"level\":\"info\",\"message\":\"User logged in\",\"metadata\":\"{\\\"user_id\\\":\\\"12345\\\",\\\"session\\\":\\\"abc-xyz\\\"}\"}",
"parsed_data": {
"level": "info",
"message": "User logged in",
"metadata": "{\"user_id\":\"12345\",\"session\":\"abc-xyz\"}"
}
},
"cache": {
"body": "{\\\"level\\\":\\\"info\\\",\\\"message\\\":\\\"User logged in\\\",\\\"metadata\\\":\\\"{\\\\\\\"user_id\\\\\\\":\\\\\\\"12345\\\\\\\",\\\\\\\"session\\\\\\\":\\\\\\\"abc-xyz\\\\\\\"}\\\"}"
}
}
The EDXUnescapeJSON function has recursively unescaped the JSON string, making it possible to properly parse the nested JSON structure.
EDXDataType
Minimum Agent Version: v1.32.0
EDXDataType fills a gap in standard OTTL which lacks type introspection capabilities. While OTTL provides type conversion functions like Int and String, it doesn’t offer a way to determine the current type of a field. This Edge Delta extension enables you to inspect data types at runtime, which is essential for debugging, conditional processing, and data validation.
Syntax
EDXDataType(input)
input: The field or value whose data type you want to determine.
Input
{
"_type": "log",
"timestamp": 1735789400000,
"body": "Test log message",
"resource": {...},
"attributes": {
"string_field": "hello",
"number_field": 42,
"float_field": 3.14,
"boolean_field": true,
"null_field": null,
"array_field": ["item1", "item2"],
"map_field": {
"key1": "value1",
"key2": "value2"
}
}
}
Example
set(attributes["types"]["string_type"], EDXDataType(attributes["string_field"]))
set(attributes["types"]["number_type"], EDXDataType(attributes["number_field"]))
set(attributes["types"]["float_type"], EDXDataType(attributes["float_field"]))
set(attributes["types"]["boolean_type"], EDXDataType(attributes["boolean_field"]))
set(attributes["types"]["null_type"], EDXDataType(attributes["null_field"]))
set(attributes["types"]["array_type"], EDXDataType(attributes["array_field"]))
set(attributes["types"]["map_type"], EDXDataType(attributes["map_field"]))
set(attributes["types"]["body_type"], EDXDataType(body))
Output
{
"_type": "log",
"timestamp": 1735789430000,
"body": "Test log message",
"resource": {...},
"attributes": {
"string_field": "hello",
"number_field": 42,
"float_field": 3.14,
"boolean_field": true,
"null_field": null,
"array_field": ["item1", "item2"],
"map_field": {
"key1": "value1",
"key2": "value2"
},
"types": {
"string_type": "string",
"number_type": "int",
"float_type": "float",
"boolean_type": "bool",
"null_type": "nil",
"array_type": "array",
"map_type": "map",
"body_type": "bytes"
}
}
}
The EDXDataType function has identified the data type of each field, returning type names such as “string”, “int”, “float”, “bool”, “nil”, “array”, “map”, and “bytes” (for the body field which is a byte array).
EDXIfElse
Minimum Agent Version: v2.0.0
EDXIfElse fills a gap in standard OTTL which lacks inline conditional operators. While OTTL supports conditional logic with where clauses for conditional execution of entire statements, it doesn’t provide a ternary operator for conditional value assignment within a single statement. This Edge Delta extension enables you to choose between two values based on a condition, similar to the ternary operator (? :) in many programming languages.
Syntax
EDXIfElse(condition, value_if_true, value_if_false)
condition: A boolean expression or field that evaluates to true or false.value_if_true: The value to return if the condition is true.value_if_false: The value to return if the condition is false.
Input
{
"_type": "log",
"timestamp": 1735789500000,
"body": "Transaction processed",
"resource": {...},
"attributes": {
"transaction_amount": 150,
"user_type": "premium",
"is_weekend": true,
"error_count": 0
}
}
Example
set(attributes["transaction_size"], EDXIfElse(attributes["transaction_amount"] > 100, "large", "small"))
set(attributes["discount_eligible"], EDXIfElse(attributes["user_type"] == "premium", true, false))
set(attributes["processing_mode"], EDXIfElse(attributes["is_weekend"], "batch", "realtime"))
set(attributes["status"], EDXIfElse(attributes["error_count"] > 0, "failed", "success"))
Output
{
"_type": "log",
"timestamp": 1735789530000,
"body": "Transaction processed",
"resource": {...},
"attributes": {
"transaction_amount": 150,
"user_type": "premium",
"is_weekend": true,
"error_count": 0,
"transaction_size": "large",
"discount_eligible": true,
"processing_mode": "batch",
"status": "success"
}
}
The EDXIfElse function has evaluated each condition and assigned the appropriate values: “large” for transaction size (amount > 100), true for discount eligibility (premium user), “batch” for processing mode (weekend), and “success” for status (no errors).
EDXCoalesce
Minimum Agent Version: v2.0.0
EDXCoalesce fills a gap in standard OTTL which lacks null coalescing capabilities. While OTTL can check for null values with IsMatch, it doesn’t provide an elegant way to select the first non-null value from multiple options. This Edge Delta extension works like the SQL COALESCE function or the null coalescing operator (??) in modern languages, enabling graceful handling of missing or empty fields with fallback values.
Syntax
EDXCoalesce(value1, value2, value3, ...)
value1, value2, value3, ...: A series of values or field references to evaluate in order. The function returns the first non-null and non-empty value.
Input
{
"_type": "log",
"timestamp": 1735789600000,
"body": "User activity log",
"resource": {...},
"attributes": {
"user_id": null,
"session_id": "",
"tracking_id": "track-12345",
"primary_email": null,
"secondary_email": "",
"fallback_email": "default@example.com",
"error_message": "",
"warning_message": null,
"info_message": "Operation completed"
}
}
Example
set(attributes["effective_id"], EDXCoalesce(attributes["user_id"], attributes["session_id"], attributes["tracking_id"], "anonymous"))
set(attributes["contact_email"], EDXCoalesce(attributes["primary_email"], attributes["secondary_email"], attributes["fallback_email"]))
set(attributes["log_message"], EDXCoalesce(attributes["error_message"], attributes["warning_message"], attributes["info_message"], "No message"))
Output
{
"_type": "log",
"timestamp": 1735789630000,
"body": "User activity log",
"resource": {...},
"attributes": {
"user_id": null,
"session_id": "",
"tracking_id": "track-12345",
"primary_email": null,
"secondary_email": "",
"fallback_email": "default@example.com",
"error_message": "",
"warning_message": null,
"info_message": "Operation completed",
"effective_id": "track-12345",
"contact_email": "default@example.com",
"log_message": "Operation completed"
}
}
The EDXCoalesce function has selected the first non-null/non-empty value for each field: “track-12345” for the effective ID (skipping null user_id and empty session_id), “default@example.com” for the contact email (after checking primary and secondary), and “Operation completed” for the log message (after checking error and warning messages).
EDXEnv
Minimum Agent Version: v2.5.0
EDXEnv fills a gap in standard OTTL which lacks environment variable access. While OTTL can work with resource attributes and other telemetry data, it cannot access system environment variables at runtime. This Edge Delta extension enables you to inject configuration values, deployment-specific information, or secrets from environment variables into your telemetry data, making pipelines more dynamic and configurable.
Syntax
EDXEnv(env_variable_name, default_value)
env_variable_name: The name of the environment variable to retrieve.default_value: The default value to return if the environment variable is not set or is empty (optional).
Input
{
"_type": "log",
"timestamp": 1735789700000,
"body": "Application started",
"resource": {...},
"attributes": {
"app_name": "telemetry-processor",
"start_time": "2024-01-10T10:00:00Z"
}
}
Assuming these environment variables are set:
ENVIRONMENT=productionREGION=us-west-2CLUSTER_NAME=telemetry-cluster-01
Example
set(attributes["environment"], EDXEnv("ENVIRONMENT", "development"))
set(attributes["region"], EDXEnv("REGION", "us-east-1"))
set(attributes["cluster"], EDXEnv("CLUSTER_NAME", "default-cluster"))
set(attributes["debug_mode"], EDXEnv("DEBUG_MODE", "false"))
set(attributes["api_version"], EDXEnv("API_VERSION", "v1"))
Output
{
"_type": "log",
"timestamp": 1735789730000,
"body": "Application started",
"resource": {...},
"attributes": {
"app_name": "telemetry-processor",
"start_time": "2024-01-10T10:00:00Z",
"environment": "production",
"region": "us-west-2",
"cluster": "telemetry-cluster-01",
"debug_mode": "false",
"api_version": "v1"
}
}
The EDXEnv function has retrieved the values from environment variables where they exist (ENVIRONMENT, REGION, CLUSTER_NAME) and used the provided default values for variables that don’t exist (DEBUG_MODE defaults to “false”, API_VERSION defaults to “v1”).
EDXExtractPatterns
Minimum Agent Version: v1.22.0
EDXExtractPatterns enhances the standard OTTL ExtractPatterns converter function. While the default OTTL function requires a hardcoded regex pattern, this Edge Delta extension enables you to use a field reference for the pattern parameter, allowing dynamic pattern matching based on runtime data.
Syntax
EDXExtractPatterns(input, "pattern")
input: The source string, such as the body of a log, from which patterns are to be extracted.pattern: A Golang regex pattern or field reference containing a regex pattern, used to match and extract specific parts of the input string. Named patterns can be used to provide field names.
Input:
{
"_type": "log",
"timestamp": 1735810699536,
"body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
"pattern": "session_id=(?P<session_id>\\w+)"
}
}
Notice how the pattern in the
patternattribute has been escaped. See Regex in OTTL.
Example:
set(attributes["id"], EDXExtractPatterns(attributes["decoded_body"], attributes["pattern"]))
Output:
{
"_type": "log",
"timestamp": 1735810732304,
"body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
"id": {
"session_id": "abc123"
},
"pattern": "session_id=(?P<session_id>\\w+)"
}
}
The pattern in the pattern attribute was used to extract the session_id’s value from the decoded body.
EDXParseKeyValue
Minimum Agent Version: v1.22.0
EDXParseKeyValue significantly enhances the standard OTTL ParseKeyValue function. While the default OTTL function overwrites duplicate keys (keeping only the last value), this Edge Delta extension provides multiple merge strategies for handling duplicate keys: keeping first, keeping last, appending as arrays, concatenating as strings, or creating indexed keys. This is essential when parsing logs that legitimately contain repeated keys.
Consider a function such as ParseKeyValue that parses key value pairs into attributes. In circumstances where a log contains multiple key-value pairs with the same key, only the last instance is used. For example:
{
"_type": "log",
"timestamp": 1735784191233,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2"
}
}
The function: set(attributes["kv_map"], ParseKeyValue(attributes["decoded_body"])), would parse the string as follows:
{
"_type": "log",
"timestamp": 1735784250481,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action": "invoice",
"location": "eu-west-2",
"service": "billing",
"status": "failure",
"user": "65532"
}
}
}
The second values for duplicate fields, such as status, have over-written the first values.
Edge Delta’s EDXParseKeyValue editor function exposes a number of options for dealing with this situation:
Syntax: EDXParseKeyValue(<target>, "<delimiter>", "<pair delimiter>", <string conversion>, "<merge strategy>")
target: the location of the field to be parsed into attributesdelimiter: The delimiter between key and value.pair delimiter: The delimiter between key-value pairs.string conversion: A Boolean value indicating whether to convert strings into float or int if a number is detected.merge strategy: Select a merge strategy: first|last|append|concat|indexed
Note: identical key-value pairs are treated as a single key-value.
Keep First
set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "first"))
In this example the first value is kept and the rest are dropped.
{
"_type": "log",
"timestamp": 1735784401372,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action": "login",
"location": "us-east-1",
"service": "auth",
"status": "success",
"user": "65532"
}
}
}
Keep Last
set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "last"))
In this example the last value is kept and the rest are dropped.
{
"_type": "log",
"timestamp": 1735784427676,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action": "invoice",
"location": "eu-west-2",
"service": "billing",
"status": "failure",
"user": "65532"
}
}
}
Append Values
set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "append"))
In this example the values are all included as an array.
{
"_type": "log",
"timestamp": 1735784499306,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action": [
"login",
"invoice"
],
"location": [
"us-east-1",
"eu-west-2"
],
"service": [
"auth",
"billing"
],
"status": [
"success",
"failure"
],
"user": "65532"
}
}
}
Concatenate Values
set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "concat"))
In this example the values are all included as a single comma separated value.
{
"_type": "log",
"timestamp": 1735784548994,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action": "login, invoice",
"location": "us-east-1, eu-west-2",
"service": "auth, billing",
"status": "success, failure",
"user": "65532"
}
}
}
Index Values
set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "indexed"))
In this example the values are all included by creating multiple indexed keys.
{
"_type": "log",
"timestamp": 1735784569792,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action_0": "login",
"action_1": "invoice",
"location_0": "us-east-1",
"location_1": "eu-west-2",
"service_0": "auth",
"service_1": "billing",
"status_0": "success",
"status_1": "failure",
"user": "65532"
}
}
}
Convert strings to int or float
set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", true, "indexed"))
In this example one string, user, is detected as being an integer and converted into the appropriate data type.
{
"_type": "log",
"timestamp": 1735784594207,
"body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"resource": {...},
"attributes": {
"decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
"kv_map": {
"action_0": "login",
"action_1": "invoice",
"location_0": "us-east-1",
"location_1": "eu-west-2",
"service_0": "auth",
"service_1": "billing",
"status_0": "success",
"status_1": "failure",
"user": 65532
}
}
}
Editor Functions
Editor functions modify existing data structures in place, such as deleting or keeping specific keys in maps.
edx_delete_keys
Minimum Agent Version: v1.23.0
edx_delete_keys enhances the standard OTTL delete_key function. While the default OTTL function can only delete one key at a time, this Edge Delta extension enables batch deletion of multiple specific keys in a single operation. This is particularly useful when you need to remove several unrelated keys that don’t follow a common pattern, eliminating the need for multiple individual delete operations.
Syntax
edx_delete_keys(input, ["key1", "key2", "key3"])
input: A map (e.g., resource or attributes) from which keys are to be deleted.keys: An array of key names to be deleted.
Input
{
"_type": "log",
"timestamp": 1735787654284,
"body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
"resource": {...},
"attributes": {
"decoded_body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
"kv_map": {
"Firewall_action": "block",
"dst_ip": "192.168.1.100",
"protocol": "TCP",
"rule_id": "R102",
"rule_name": "Block_All_Outgoing",
"severity": "high",
"src_ip": "10.0.0.1",
"user_group": "network_admins",
"user_id": "admin"
}
}
}
Example
edx_delete_keys(attributes["kv_map"], ["rule_id", "rule_name", "user_id", "user_group"])
Output
{
"_type": "log",
"timestamp": 1735787684654,
"body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
"resource": {...},
"attributes": {
"decoded_body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
"kv_map": {
"Firewall_action": "block",
"dst_ip": "192.168.1.100",
"protocol": "TCP",
"severity": "high",
"src_ip": "10.0.0.1"
}
}
}
The keys rule_id, rule_name, user_id, and user_group have been removed as a batch in a single operation.
edx_delete_matching_keys
Minimum Agent Version: v1.23.0
edx_delete_matching_keys enhances the standard OTTL delete_matching_keys function. While the default OTTL function accepts only a single regex pattern, this Edge Delta extension enables you to provide multiple regex patterns in a single operation. This allows concurrent deletion of keys matching different patterns without requiring multiple separate statements.
Syntax
edx_delete_matching_keys(input, ["key1", "key2"])
input: A map (e.g., resource or attributes) from which keys matching specified patterns are to be deleted.keys: An array of regex-like patterns for matching and deleting keys.
Input
{
"_type": "log",
"timestamp": 1735788401264,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"connection_id": "conn456",
"debug_info": "verbose",
"event_status": "success",
"event_type": "login",
"session_id": "abc123",
"temp_data": "sensitiveData",
"temp_file": "report_tmp.log",
"user_id": "admin"
}
}
}
Example
edx_delete_matching_keys(attributes["kv_map"], ["(.*_id$)", "(temp_.*|debug_.*)"])
Output
{
"_type": "log",
"timestamp": 1735788441183,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"event_status": "success",
"event_type": "login"
}
}
}
Keys matching the two patterns (.*_id$) (connection_id, session_id, user_id) and (temp_.*|debug_.*) (debug_info, temp_data, temp_file) were removed.
edx_delete_empty_values
Minimum Agent Version: v1.28.0
edx_delete_empty_values fills a gap in standard OTTL which lacks a dedicated function for cleaning up empty or meaningless values. While OTTL can delete specific keys with delete_key, it doesn’t provide a way to automatically remove all keys with empty, null, or custom-defined “empty” values (like “unknown” or “N/A”). This Edge Delta extension enables comprehensive data cleanup by removing fields that contain no meaningful information based on customizable criteria.
Syntax
edx_delete_empty_values(input, excluded_keys, empty_patterns, options)
input: A map (e.g., attributes, resource) from which empty values should be deleted.excluded_keys: An array of key names to exclude from deletion (optional, use[]to exclude none).empty_patterns: An array of string patterns to consider as empty values (e.g.,["", "unknown", "N/A"]).options: An array of options for additional deletion behavior (e.g.,["deleteNull"]to also delete null values).
Input
{
"_type": "log",
"timestamp": 1735788500000,
"body": "user_id=12345 name=John email= status=unknown location= department=null notes=",
"resource": {...},
"attributes": {
"decoded_body": "user_id=12345 name=John email= status=unknown location= department=null notes=",
"kv_map": {
"user_id": "12345",
"name": "John",
"email": "",
"status": "unknown",
"location": "",
"department": null,
"notes": ""
}
}
}
Example
edx_delete_empty_values(attributes["kv_map"], [], ["", "unknown"], ["deleteNull"])
Output
{
"_type": "log",
"timestamp": 1735788530000,
"body": "user_id=12345 name=John email= status=unknown location= department=null notes=",
"resource": {...},
"attributes": {
"decoded_body": "user_id=12345 name=John email= status=unknown location= department=null notes=",
"kv_map": {
"user_id": "12345",
"name": "John"
}
}
}
The keys email, status, location, department, and notes have been removed because their values were either empty strings, matched the pattern “unknown”, or were null (with the deleteNull option enabled).
edx_keep_keys
Minimum Agent Version: v1.23.0
edx_keep_keys enhances the standard OTTL keep_keys function. While the default OTTL function requires individual key specifications, this Edge Delta extension accepts an array of keys to retain in a single batch operation. This is more efficient when you need to keep multiple specific keys and remove everything else, essentially providing an allowlist approach to field management.
Syntax
edx_keep_keys(input, ["key1", "key2"])
input: A map (e.g., resource or attributes) from which keys are selectively retained.keys: An array of key names to be retained.
Input:
{
"_type": "log",
"timestamp": 1735796321870,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"connection_id": "conn456",
"debug_info": "verbose",
"event_status": "success",
"event_type": "login",
"session_id": "abc123",
"temp_data": "sensitiveData",
"temp_file": "report_tmp.log",
"user_id": "admin"
}
}
}
Example:
edx_keep_keys(attributes["kv_map"], ["event_status", "event_type"])
Output:
{
"_type": "log",
"timestamp": 1735796349262,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"event_status": "success",
"event_type": "login"
}
}
}
Only the keys event_status and event_type have been retained in the kv_map map.
edx_keep_matching_keys
Minimum Agent Version: v1.23.0
edx_keep_matching_keys enhances the standard OTTL keep_matching_keys function. While the default OTTL function accepts only a single regex pattern, this Edge Delta extension enables you to provide multiple regex patterns simultaneously. This allows you to retain keys matching any of several patterns in one operation, making complex filtering scenarios more efficient.
Syntax
edx_keep_matching_keys(input, ["key1", "key2"])
input: A map (e.g., resource or attributes) from which keys are selectively retained.keys: An array of regex-like patterns for matching and retaining keys.
Input:
{
"_type": "log",
"timestamp": 1735796604731,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"connection_id": "conn456",
"debug_info": "verbose",
"event_status": "success",
"event_type": "login",
"session_id": "abc123",
"temp_data": "sensitiveData",
"temp_file": "report_tmp.log",
"user_id": "admin"
}
}
}
Example:
edx_keep_matching_keys(attributes["kv_map"], ["(.*_id$)", "(temp_.*|debug_.*)"])
Output:
{
"_type": "log",
"timestamp": 1735796570758,
"body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"resource": {...},
"attributes": {
"decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
"kv_map": {
"connection_id": "conn456",
"debug_info": "verbose",
"session_id": "abc123",
"temp_data": "sensitiveData",
"temp_file": "report_tmp.log",
"user_id": "admin"
}
}
}
The keys event_status and event_type have been removed as they do not match one of the patterns.
edx_map_keys
Minimum Agent Version: v1.26.0
edx_map_keys fills a gap in standard OTTL which lacks bulk field renaming capabilities. While OTTL can set and delete individual fields to achieve renaming, it doesn’t provide a way to transform multiple field names according to a schema mapping in a single operation. This Edge Delta extension enables schema normalization by renaming multiple keys at once, which is essential for adapting data to different standards or destination requirements.
Syntax
edx_map_keys(input, current_keys, new_keys, strategy)
input: A map (e.g., attributes, resource) whose keys should be mapped to new names.current_keys: An array of current key names to be mapped from.new_keys: An array of new key names to map to (must have the same length as current_keys).strategy: The mapping strategy to use (e.g., “update” to rename keys).
Input
{
"_type": "log",
"timestamp": 1735789000000,
"body": "src_ip=10.0.0.1 dst_ip=192.168.1.100 proto=TCP action=ALLOW bytes=1024",
"resource": {...},
"attributes": {
"curr_schema": ["src_ip", "dst_ip", "proto", "action", "bytes"],
"new_schema": ["source_address", "destination_address", "protocol", "firewall_action", "byte_count"],
"decoded_body": "src_ip=10.0.0.1 dst_ip=192.168.1.100 proto=TCP action=ALLOW bytes=1024",
"kv_map": {
"src_ip": "10.0.0.1",
"dst_ip": "192.168.1.100",
"proto": "TCP",
"action": "ALLOW",
"bytes": "1024"
}
}
}
Example
edx_map_keys(attributes["kv_map"], attributes["curr_schema"], attributes["new_schema"], "update")
Output
{
"_type": "log",
"timestamp": 1735789030000,
"body": "src_ip=10.0.0.1 dst_ip=192.168.1.100 proto=TCP action=ALLOW bytes=1024",
"resource": {...},
"attributes": {
"curr_schema": ["src_ip", "dst_ip", "proto", "action", "bytes"],
"new_schema": ["source_address", "destination_address", "protocol", "firewall_action", "byte_count"],
"decoded_body": "src_ip=10.0.0.1 dst_ip=192.168.1.100 proto=TCP action=ALLOW bytes=1024",
"kv_map": {
"source_address": "10.0.0.1",
"destination_address": "192.168.1.100",
"protocol": "TCP",
"firewall_action": "ALLOW",
"byte_count": "1024"
}
}
}
The keys in kv_map have been renamed according to the mapping defined in curr_schema and new_schema. Each key from the current schema has been replaced with its corresponding key from the new schema, enabling seamless schema transformation for data normalization or integration with different systems.