Edge Delta's OTTL Extensions

Transform or route data using Edge Delta’s OTTL extensions.

  9 minute read  

Beta

Edge Delta Custom OTTL

Edge Delta has extended the OTTL with custom functionality.

EDXEncode

When a log is ingested by the Edge Delta agent, a log data item is created and the log contents are placed in the body field. Importantly, this field is a byte array. Therefore, to perform any OTTL transformation that requires the body as a string input, it needs to be decoded from byte array to a string type as part of the transformation function. See Working with the body.

After copying, decoding and transforming the body into a field other than body, you can use the EDXEncode custom function to save a new body field in the appropriate byte array type.

Syntax

EDXEncode(target_field, current_format, target_encoding)
  • target_field: A field that should be converted to byte array.
  • current_format: The current type from the IANA encoding index, such as “utf-8”; or “base64”.
  • target_encoding: A boolean for the target type: true for byte array, false for string.

Suppose this log is ingested:

session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData

The following operations are performed to extract, decode, parse, and transform the data:

set(attributes["decoded_body"], Decode(body, "utf-8"))
set(attributes["kv_map"], ParseKeyValue(attributes["decoded_body"]))
delete_matching_keys(attributes["kv_map"], "(temp_.*|debug_.*)")
set(attributes["new_body"], ToKeyValueString(attributes["kv_map"]))

See Decode, ParseKeyValue, delete_matching_keys, and ToKeyValueString.

These operations result in the following log:

Input:

{
  "_type": "log",
  "timestamp": 1735802408445,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "connection_id": "conn456",
      "event_status": "success",
      "event_type": "login",
      "session_id": "abc123",
      "user_id": "admin"
    },
    "new_body": "event_status=success connection_id=conn456 session_id=abc123 user_id=admin event_type=login"
  }
}

Example:

set(body, EDXEncode(attributes["new_body"], "utf-8", true))

Output:

{
  "_type": "log",
  "timestamp": 1735802426292,
  "body": "connection_id=conn456 session_id=abc123 user_id=admin event_type=login event_status=success",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "connection_id": "conn456",
      "event_status": "success",
      "event_type": "login",
      "session_id": "abc123",
      "user_id": "admin"
    },
    "new_body": "connection_id=conn456 session_id=abc123 user_id=admin event_type=login event_status=success"
  }
}

The new_body attribute has been upserted into the body field and encoded as a byte array.

EDXExtractPatterns

EDXExtractPatterns is a converter function used for extracting values using a regex pattern. While the default OTTL ExtractPatterns converter function can extract values using a regex pattern, this Edge Delta extension enables you to use a field reference for the pattern parameter.

Syntax

EDXExtractPatterns(input, "pattern")
  • input: The source string, such as the body of a log, from which patterns are to be extracted.
  • pattern: A Golang regex pattern or field reference containing a regex pattern, used to match and extract specific parts of the input string. Named patterns can be used to provide field names.

Input:

{
  "_type": "log",
  "timestamp": 1735810699536,
  "body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
    "pattern": "session_id=(?P<session_id>\\w+)"
  }
}

Notice how the pattern in the pattern attribute has been escaped. See Regex in OTTL.

Example:

set(attributes["id"], EDXExtractPatterns(attributes["decoded_body"], attributes["pattern"]))

Output:

{
  "_type": "log",
  "timestamp": 1735810732304,
  "body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 event_type=login event_status=success debug_info=verbose temp_file=report_tmp.log temp_data=sensitiveData",
    "id": {
      "session_id": "abc123"
    },
    "pattern": "session_id=(?P<session_id>\\w+)"
  }
}

The pattern in the pattern attribute was used to extract the session_id’s value from the decoded body.

EDXParseKeyValue

Consider a function such as ParseKeyValue that parses key value pairs into attributes. In circumstances where a log contains multiple key-value pairs with the same key, only the last instance is used. For example:

{
  "_type": "log",
  "timestamp": 1735784191233,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2"
  }
}

The function: set(attributes["kv_map"], ParseKeyValue(attributes["decoded_body"])), would parse the string as follows:

{
  "_type": "log",
  "timestamp": 1735784250481,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "invoice",
      "location": "eu-west-2",
      "service": "billing",
      "status": "failure",
      "user": "65532"
    }
  }
}

The second values for duplicate fields, such as status, have over-written the first values.

Edge Delta’s EDXParseKeyValue editor function exposes a number of options for dealing with this situation:

Syntax: EDXParseKeyValue(<target>, "<delimiter>", "<pair delimiter>", <string conversion>, "<merge strategy>")

  • target: the location of the field to be parsed into attributes
  • delimiter: The delimiter between key and value.
  • pair delimiter: The delimiter between key-value pairs.
  • string conversion: A Boolean value indicating whether to convert strings into float or int if a number is detected.
  • merge strategy: Select a merge strategy: first|last|append|concat|indexed

Note: identical key-value pairs are treated as a single key-value.

Keep First

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "first"))

In this example the first value is kept and the rest are dropped.

{
  "_type": "log",
  "timestamp": 1735784401372,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "login",
      "location": "us-east-1",
      "service": "auth",
      "status": "success",
      "user": "65532"
    }
  }
}

Keep Last

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "last"))

In this example the last value is kept and the rest are dropped.

{
  "_type": "log",
  "timestamp": 1735784427676,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "invoice",
      "location": "eu-west-2",
      "service": "billing",
      "status": "failure",
      "user": "65532"
    }
  }
}

Append Values

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "append"))

In this example the values are all included as an array.

{
  "_type": "log",
  "timestamp": 1735784499306,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": [
        "login",
        "invoice"
      ],
      "location": [
        "us-east-1",
        "eu-west-2"
      ],
      "service": [
        "auth",
        "billing"
      ],
      "status": [
        "success",
        "failure"
      ],
      "user": "65532"
    }
  }
}

Concatenate Values

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "concat"))

In this example the values are all included as a single comma separated value.

{
  "_type": "log",
  "timestamp": 1735784548994,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "login, invoice",
      "location": "us-east-1, eu-west-2",
      "service": "auth, billing",
      "status": "success, failure",
      "user": "65532"
    }
  }
}

Index Values

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "indexed"))

In this example the values are all included by creating multiple indexed keys.

{
  "_type": "log",
  "timestamp": 1735784569792,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action_0": "login",
      "action_1": "invoice",
      "location_0": "us-east-1",
      "location_1": "eu-west-2",
      "service_0": "auth",
      "service_1": "billing",
      "status_0": "success",
      "status_1": "failure",
      "user": "65532"
    }
  }
}

Convert strings to int or float

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", true, "indexed"))

In this example one string, user, is detected as being an integer and converted into the appropriate data type.

{
  "_type": "log",
  "timestamp": 1735784594207,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action_0": "login",
      "action_1": "invoice",
      "location_0": "us-east-1",
      "location_1": "eu-west-2",
      "service_0": "auth",
      "service_1": "billing",
      "status_0": "success",
      "status_1": "failure",
      "user": 65532
    }
  }
}

edx_delete_keys

The edx_delete_keys editor function deletes specified keys from the provided map using an array of key names. This capability enhances the delete_key function by allowing batch deletion of multiple specific keys in a single operation, and where the keys do not follow a common regex pattern.

Syntax

edx_delete_keys(input, ["key1", "key2", "key3"])
  • input: A map (e.g., resource or attributes) from which keys are to be deleted.
  • keys: An array of key names to be deleted.

Input

{
  "_type": "log",
  "timestamp": 1735787654284,
  "body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
  "resource": {...},
  "attributes": {
    "decoded_body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
    "kv_map": {
      "Firewall_action": "block",
      "dst_ip": "192.168.1.100",
      "protocol": "TCP",
      "rule_id": "R102",
      "rule_name": "Block_All_Outgoing",
      "severity": "high",
      "src_ip": "10.0.0.1",
      "user_group": "network_admins",
      "user_id": "admin"
    }
  }
}

Example

edx_delete_keys(attributes["kv_map"], ["rule_id", "rule_name", "user_id", "user_group"])

Output

{
  "_type": "log",
  "timestamp": 1735787684654,
  "body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
  "resource": {...},
  "attributes": {
    "decoded_body": "Firewall_action=block rule_id=R102 rule_name=Block_All_Outgoing user_id=admin user_group=network_admins protocol=TCP severity=high src_ip=10.0.0.1 dst_ip=192.168.1.100",
    "kv_map": {
      "Firewall_action": "block",
      "dst_ip": "192.168.1.100",
      "protocol": "TCP",
      "severity": "high",
      "src_ip": "10.0.0.1"
    }
  }
}

The keys rule_id, rule_name, user_id, and user_group have been removed as a batch in a single operation.

edx_delete_matching_keys

The edx_delete_matching_keys editor function deletes keys that match specified patterns from the provided map. It extends the delete_matching_keys functionality by enabling the concurrent deletion of keys from a map using multiple regex patterns in a single operation.

Syntax

edx_delete_matching_keys(input, ["key1", "key2"])
  • input: A map (e.g., resource or attributes) from which keys matching specified patterns are to be deleted.
  • keys: An array of regex-like patterns for matching and deleting keys.

Input

{
  "_type": "log",
  "timestamp": 1735788401264,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "connection_id": "conn456",
      "debug_info": "verbose",
      "event_status": "success",
      "event_type": "login",
      "session_id": "abc123",
      "temp_data": "sensitiveData",
      "temp_file": "report_tmp.log",
      "user_id": "admin"
    }
  }
}

Example

edx_delete_matching_keys(attributes["kv_map"], ["(.*_id$)", "(temp_.*|debug_.*)"])

Output

{
  "_type": "log",
  "timestamp": 1735788441183,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "event_status": "success",
      "event_type": "login"
    }
  }
}

Keys matching the two patterns (.*_id$) (connection_id, session_id, user_id) and (temp_.*|debug_.*) (debug_info, temp_data, temp_file) were removed.

edx_keep_keys

Overview: The edx_keep_keys editor function retains only the keys explicitly listed in its parameters, offering a straightforward approach to manage datasets by excluding all unspecified keys. It extends the OTTL OOTB keep_keys function by supporting a single batch operation from a provided map.

Syntax

edx_keep_keys(input, ["key1", "key2"])
  • input: A map (e.g., resource or attributes) from which keys are selectively retained.
  • keys: An array of key names to be retained.

Input:

{
  "_type": "log",
  "timestamp": 1735796321870,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "connection_id": "conn456",
      "debug_info": "verbose",
      "event_status": "success",
      "event_type": "login",
      "session_id": "abc123",
      "temp_data": "sensitiveData",
      "temp_file": "report_tmp.log",
      "user_id": "admin"
    }
  }
}

Example:

edx_keep_keys(attributes["kv_map"], ["event_status", "event_type"])

Output:

{
  "_type": "log",
  "timestamp": 1735796349262,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "event_status": "success",
      "event_type": "login"
    }
  }
}

Only the keys event_status and event_type have been retained in the kv_map map.

edx_keep_matching_keys

Overview: The edx_keep_matching_keys editor function retains keys that match specified regex-like patterns, supporting dynamic data management by focusing on keys that fit defined contextual patterns. It extends keep_matching_keys by facilitating the simultaneous use of multiple Golang Regex patterns to retain keys.

Syntax

edx_keep_matching_keys(input, ["key1", "key2"])
  • input: A map (e.g., resource or attributes) from which keys are selectively retained.
  • keys: An array of regex-like patterns for matching and retaining keys.

Input:

{
  "_type": "log",
  "timestamp": 1735796604731,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "connection_id": "conn456",
      "debug_info": "verbose",
      "event_status": "success",
      "event_type": "login",
      "session_id": "abc123",
      "temp_data": "sensitiveData",
      "temp_file": "report_tmp.log",
      "user_id": "admin"
    }
  }
}

Example:

edx_keep_matching_keys(attributes["kv_map"], ["(.*_id$)", "(temp_.*|debug_.*)"])

Output:

{
  "_type": "log",
  "timestamp": 1735796570758,
  "body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
  "resource": {...},
  "attributes": {
    "decoded_body": "session_id=abc123 user_id=admin event_type=login event_status=success debug_info=verbose connection_id=conn456 temp_file=report_tmp.log temp_data=sensitiveData",
    "kv_map": {
      "connection_id": "conn456",
      "debug_info": "verbose",
      "session_id": "abc123",
      "temp_data": "sensitiveData",
      "temp_file": "report_tmp.log",
      "user_id": "admin"
    }
  }
}

The keys event_status and event_type have been removed as they do not match one of the patterns.