Edge Delta's OTTL Extensions

Transform or route data using Edge Delta’s OTTL extensions.

  6 minute read  

Beta

Edge Delta Custom OTTL

Edge Delta has extended the OTTL with additional custom functionality.

EDXParseKeyValue

Consider a function that parses key value pairs into attributes. In circumstances where a log contains multiple key-value pairs with the same key, traditionally only the last instance is used. For example:

{
  "body": "key1=1.1|key1=2|key1=three"
}

would be parsed as

{
  "attributes": {
    "key1": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

The values 1.1 and 2 have been lost.

Edge Delta’s EDXParseKeyValue editor function exposes a number of options for dealing with this situation:

Syntax: EDXParseKeyValue(<target>, "<delimiter>", "<pair delimiter>", <string conversion>, "<merge strategy>")

  • target: the location of the field to be parsed into attributes
  • delimiter: The delimiter between key and value.
  • pair delimiter: The delimiter between key-value pairs.
  • string conversion: A Boolean value indicating whether to convert strings into float or int if a number is detected.
  • merge strategy: Select a merge strategy: first|last|append|concat|indexed

Note: identical key-value pairs are treated as a single key-value.

Examples:

Consider this input:

{
  "body": "key1=1.1|key1=2|key1=three"
}

Keep First

set(attributes, EDXParseKeyValue(body, "=", "|", false, "first"))

In this example the first value is kept and the rest are dropped.

{
  "attributes": {
    "key1": "1.1"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Keep Last

set(attributes, EDXParseKeyValue(body, "=", "|", false, "last"))

In this example the last value is kept and the rest are dropped.

{
  "attributes": {
    "key1": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Append Values

set(attributes, EDXParseKeyValue(body, "=", "|", false, "append"))

In this example the values are all included as an array.

{
  "attributes": {
    "key1": [
      "1.1",
      "2",
      "three"
    ]
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Concatenate Values

set(attributes, EDXParseKeyValue(body, "=", "|", false, "concat"))

In this example the values are all included as a single comma separated value.

{
  "attributes": {
    "key1": "1.1, 2, three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Index Values

set(attributes, EDXParseKeyValue(body, "=", "|", false, "indexed"))

In this example the values are all included by creating multiple indexed keys.

{
  "attributes": {
    "key1_0": "1.1",
    "key1_1": "2",
    "key1_2": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

Convert strings to int or float

set(attributes, EDXParseKeyValue(body, "=", "|", true, "indexed"))

In this example some strings are detected as being an integer and a float and converted into the appropriate data types. The string remains unchanged.

{
  "attributes": {
    "key1_0": 1.1,
    "key1_1": 2,
    "key1_2": "three"
  },
  "body": "key1=1.1|key1=2|key1=three"
}

edx_delete_keys

The edx_delete_keys editor function deletes specified keys from the provided map using an array of key names. This capability enhances the typical OOTB delete_key function by allowing batch deletion of multiple specific keys in a single operation. Thus, it simplifies code and improves script performance in environments where multiple keys are routinely managed.

Syntax

edx_delete_keys(input, ["key1", "key2", "key3"])
  • input: A map (e.g., resource or attributes) from which keys are to be deleted.
  • keys: An array of key names to be deleted.

Example

edx_delete_keys(resource, ["host.ip", "src_type", "service.name"])

Input

{
  "_type": "log",
  "body": "time=hostname=SIMPLE|product=Firewall",
  "resource": {
    "host.ip": "192.168.1.1",
    "host.name": "simple-host",
    "service.name": "basic-service",
    "src_type": "http",
    "host.extra": "extra-info"
  },
  "timestamp": 1730511053177
}

Output

{
  "_type": "log",
  "body": "time=hostname=SIMPLE|product=Firewall",
  "resource": {
    "host.name": "simple-host",
    "host.extra": "extra-info"
  },
  "timestamp": 1730511053177
}

The keys host.ip, src_type, and service.name have been removed from the resource map.

edx_delete_matching_keys

The edx_delete_matching_keys editor function deletes keys that match specified patterns from the provided map. By handling multiple patterns in a single call, it optimizes log processing in structured environments such as Kubernetes. It extends the delete_matching_keys functionality by enabling the concurrent deletion of keys from a map using multiple regex-like patterns in a single call.

Syntax

edx_delete_matching_keys(input, ["key1", "key2"])
  • input: A map (e.g., resource or attributes) from which keys matching specified patterns are to be deleted.
  • keys: An array of regex-like patterns for matching and deleting keys.

Example

edx_delete_matching_keys(resource, ["__.*", "k8s\.pod\..*"])

Input

{
  "_type": "log",
  "body": "time=hostname=SIMPLE|product=Firewall",
  "resource": {
    "host.name": "simple-host",
    "__src_name": "source_name",
    "k8s.pod.name": "api-pod-1234",
    "service.name": "basic-service"
  },
  "timestamp": 1730511053177
}

Output

{
  "_type": "log",
  "body": "time=hostname=SIMPLE|product=Firewall",
  "resource": {
    "host.name": "simple-host",
    "service.name": "basic-service"
  },
  "timestamp": 1730511053177
}

Keys matching the patterns __.* and k8s.pod.* were removed, while keys like host.name and service.name were retained, illustrating effective pattern-matched deletion in a Kubernetes context.

edx_keep_keys

Overview: The edx_keep_keys editor function retains only the keys explicitly listed in its parameters, offering a straightforward approach to manage datasets by excluding all unspecified keys. It extends the OTTL OOTB keep_keys function by supporting a single batch operation from a provided map.

Syntax

edx_keep_keys(input, ["key1", "key2"])
  • input: A map (e.g., resource or attributes) from which keys are selectively retained.
  • keys: An array of key names to be retained.

Example:

edx_keep_keys(resource, ["os.type", "service.name"])

Input:

{
  "_type": "log",
  "body": "device=Computer|status=Active|role=Backend",
  "resource": {
    "location": "DataCenterA",
    "os.type": "Linux",
    "service.name": "api-service",
    "environment": "production",
    "team": "backend"
  },
  "timestamp": 1730511053177
}

Output:

{
  "_type": "log",
  "body": "device=Computer|status=Active|role=Backend",
  "resource": {
    "os.type": "Linux",
    "service.name": "api-service"
  },
  "timestamp": 1730511053177
}

Only the keys os.type and service.name have been retained in the resource map, successfully excluding all other keys and demonstrating precise key retention based on the specified parameters.

edx_keep_matching_keys

Overview: The edx_keep_matching_keys editor function retains keys that match specified regex-like patterns, supporting dynamic data management by focusing on keys that fit defined contextual patterns. It extends keep_matching_keys by facilitating the simultaneous use of multiple Golang Regex patterns to retain keys.

Syntax

edx_keep_matching_keys(input, ["key1", "key2"])
  • input: A map (e.g., resource or attributes) from which keys are selectively retained.
  • keys: An array of regex-like patterns for matching and retaining keys.

Example:

edx_keep_matching_keys(resource, ["^os\..*", "team.*"])

Input:

{
  "_type": "log",
  "body": "device=Computer|status=Active|role=Backend",
  "resource": {
    "location": "DataCenterA",
    "os.type": "Linux",
    "service.name": "api-service",
    "environment": "production",
    "team": "backend"
  },
  "timestamp": 1730511053177
}

Output:

{
  "_type": "log",
  "body": "device=Computer|status=Active|role=Backend",
  "resource": {
    "os.type": "Linux",
    "team": "backend"
  },
  "timestamp": 1730511053177
}

The keys that match the patterns ^os\..* and team.* have been retained in the resource map, while all other keys have been removed.

EDXExtractPatterns

EDXExtractPatterns is a converter function used for extracting specific patterns from log data. It enables the isolation of nested or detailed components within logs using Golang regex patterns, facilitating structured data manipulation and analysis.

Syntax

EDXExtractPatterns(input, "pattern")
  • input: The source string, such as the body of a log, from which patterns are to be extracted.
  • pattern: A Golang regex pattern used to match and extract specific parts of the input string. Named patterns can be employed for clear data retrieval.

Example:

set(attributes["extracted"], EDXExtractPatterns(body, ""target_labels": "{(?P<inside>[^}]+)}""))

Input:

{
  "_type": "log",
  "body": "{"action": "update","target_labels": "{key1=value1,key2=value2}","status": "completed"}",
  "timestamp": 1730511053177
}

Output:

{
  "_type": "log",
  "attributes": {
    "extracted": {
      "inside": "key1=value1,key2=value2"
    }
  },
  "body": "{"action": "update","target_labels": "{key1=value1,key2=value2}","status": "completed"}",
  "timestamp": 1730511053177
}

The function captures the content of target_labels, leveraging Golang regex patterns with named groups to structure the extracted data for subsequent processing.

Following extraction, operators like replace_pattern and ParseKeyValue can further manipulate the data, allowing for comprehensive analysis, transformation, or storage operations.