EDXParseKeyValue

Learn about the EDXParseKeyValue Edge Delta OTTL extension function.

  4 minute read  

Minimum Agent Version: v1.22.0

EDXParseKeyValue significantly enhances the standard OTTL ParseKeyValue function. While the default OTTL function overwrites duplicate keys (keeping only the last value), this Edge Delta extension provides multiple merge strategies for handling duplicate keys: keeping first, keeping last, appending as arrays, concatenating as strings, or creating indexed keys. This is essential when parsing logs that legitimately contain repeated keys.

Consider a function such as ParseKeyValue that parses key value pairs into attributes. In circumstances where a log contains multiple key-value pairs with the same key, only the last instance is used. For example:

{
  "_type": "log",
  "timestamp": 1735784191233,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2"
  }
}

The function: set(attributes["kv_map"], ParseKeyValue(attributes["decoded_body"])), would parse the string as follows:

{
  "_type": "log",
  "timestamp": 1735784250481,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "invoice",
      "location": "eu-west-2",
      "service": "billing",
      "status": "failure",
      "user": "65532"
    }
  }
}

The second values for duplicate fields, such as status, have over-written the first values.

Edge Delta’s EDXParseKeyValue editor function exposes a number of options for dealing with this situation:

Syntax: EDXParseKeyValue(<target>, "<delimiter>", "<pair delimiter>", <string conversion>, "<merge strategy>")

  • target: the location of the field to be parsed into attributes
  • delimiter: The delimiter between key and value.
  • pair delimiter: The delimiter between key-value pairs.
  • string conversion: A Boolean value indicating whether to convert strings into float or int if a number is detected.
  • merge strategy: Select a merge strategy: first|last|append|concat|indexed

Note: identical key-value pairs are treated as a single key-value.

Keep First

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "first"))

In this example the first value is kept and the rest are dropped.

{
  "_type": "log",
  "timestamp": 1735784401372,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "login",
      "location": "us-east-1",
      "service": "auth",
      "status": "success",
      "user": "65532"
    }
  }
}

Keep Last

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "last"))

In this example the last value is kept and the rest are dropped.

{
  "_type": "log",
  "timestamp": 1735784427676,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "invoice",
      "location": "eu-west-2",
      "service": "billing",
      "status": "failure",
      "user": "65532"
    }
  }
}

Append Values

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "append"))

In this example the values are all included as an array.

{
  "_type": "log",
  "timestamp": 1735784499306,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": [
        "login",
        "invoice"
      ],
      "location": [
        "us-east-1",
        "eu-west-2"
      ],
      "service": [
        "auth",
        "billing"
      ],
      "status": [
        "success",
        "failure"
      ],
      "user": "65532"
    }
  }
}

Concatenate Values

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "concat"))

In this example the values are all included as a single comma separated value.

{
  "_type": "log",
  "timestamp": 1735784548994,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action": "login, invoice",
      "location": "us-east-1, eu-west-2",
      "service": "auth, billing",
      "status": "success, failure",
      "user": "65532"
    }
  }
}

Index Values

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", false, "indexed"))

In this example the values are all included by creating multiple indexed keys.

{
  "_type": "log",
  "timestamp": 1735784569792,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action_0": "login",
      "action_1": "invoice",
      "location_0": "us-east-1",
      "location_1": "eu-west-2",
      "service_0": "auth",
      "service_1": "billing",
      "status_0": "success",
      "status_1": "failure",
      "user": "65532"
    }
  }
}

Convert strings to int or float

set(attributes["kv_map"], EDXParseKeyValue(attributes["decoded_body"], "=", " ", true, "indexed"))

In this example one string, user, is detected as being an integer and converted into the appropriate data type.

{
  "_type": "log",
  "timestamp": 1735784594207,
  "body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
  "resource": {...},
  "attributes": {
    "decoded_body": "service=auth user=65532 action=login status=success location=us-east-1 service=billing action=invoice status=failure location=eu-west-2",
    "kv_map": {
      "action_0": "login",
      "action_1": "invoice",
      "location_0": "us-east-1",
      "location_1": "eu-west-2",
      "service_0": "auth",
      "service_1": "billing",
      "status_0": "success",
      "status_1": "failure",
      "user": 65532
    }
  }
}