Edge Delta OTTL Transform Node
5 minute read
Overview
OTTL (Observability Telemetry Transformation Language) statements are used for transforming telemetry data. OTTL allows specifying transformations and operations on the data, using a defined language structure and functions. You can use it for altering field values, upserting (update or insert) new fields, and applying conditional logic to modify the data set based on certain criteria.
The OTTL node enables you to transform data using OTTL statements.
Note: advanced transformations may have an impact on agent performance.
OTTL Statements
See the OpenTelemetry OTTL documentation for more details about the OTTL.
Upserting (Update or Insert) New Fields with a Static Value
Syntax: set(target, value)
Here, the new_field
attribute is added with a static value “defaultValue
” if it doesn’t exist, or updated if it does.
set(attributes["new_field"], "defaultValue")
Converting the Case
Syntax: set(target, value)
Syntax: ConvertCase(target, toCase)
This statement converts the value of existing_field
to upper case and assigns it to a new attribute capitalized_field
.
set(attributes["capitalized_field"], ConvertCase(attributes["existing_field"], "upper"))
Decoding a Base64 String
Syntax: set(target, value)
Syntax: Decode(value, encoding)
This example decodes a base64 string and assigns the result to a base64
attribute.
set(attributes["base64"], Decode("aGVsbG8gd29ybGQ=", "base64"))
Applying Conditional Logic to Modify the Data Set
Syntax: syntax: set(target, value) where <condition>
This statement sets the alert field to Critical
only if the status field is error
, applying conditional logic for field modification.
set(attributes["alert"], "Critical") where attributes["status"] == "error"
Appending Tags
Syntax: append(target, value)
This example appends a new tag to the tags
attribute list.
append(attributes["tags"], "prod")
Deleting a Specific Key
Syntax: delete_key(target, key)
This example removes a specific key from attributes.
delete_key(attributes, "http.request.header.authorization")
Deleting Matching Keys
Syntax: delete_matching_keys(target, pattern)
Use this to remove keys that match a particular regex pattern, like sensitive keys.
delete_matching_keys(attributes, "(?i).*password.*")
Keeping Only Matching Keys
Syntax: keep_matching_keys(target, pattern)
This keeps keys that match a given pattern, filtering out all other keys.
keep_matching_keys(attributes, "(?i).*version.*")
Flattening Nested Structures:
Syntax: flatten(target)
This is used to flatten nested attribute structures.
flatten(attributes)
Setting a Duration from a String
Syntax: set(target, value)
Syntax: Duration(value)
Syntax: where
This converts a string representation of a duration into a duration object if the original value is a string.
set(attributes["duration"], Duration(attributes["duration_str"])) where IsString(attributes["duration_str"])
Edge Delta Custom OTTL Statements
Edge Delta has extended the OTTL with additional custom functionality.
EDXParseKeyValue
Consider a function that parses key value pairs into attributes. In circumstances where a log contains multiple key-value pairs with the same key, traditionally only the last instance is used. For example:
{
"body": "key1=1.1|key1=2|key1=three"
}
would be parsed as
{
"attributes": {
"key1": "three"
},
"body": "key1=1.1|key1=2|key1=three"
}
The values 1.1 and 2 have been lost.
Edge Delta’s EDXParseKeyValue
function exposes a number of options for dealing with this situation:
Syntax: EDXParseKeyValue(<target>, "<delimiter>", "<pair delimiter>", <string conversion>, "<merge strategy>")
target
: the location of the field to be parsed into attributesdelimiter
: The delimiter between key and value.pair delimiter
: The delimiter between key-value pairs.string conversion
: A Boolean value indicating whether to convert strings into float or int if a number is detected.merge strategy
: Select a merge strategy: first|last|append|concat|indexed
Note: identical key-value pairs are treated as a single key-value.
Examples:
Consider this input:
{
"body": "key1=1.1|key1=2|key1=three"
}
Keep First
set(attributes, EDXParseKeyValue(body, "=", "|", false, "first"))
In this example the first value is kept and the rest are dropped.
{
"attributes": {
"key1": "1.1"
},
"body": "key1=1.1|key1=2|key1=three"
}
Keep Last
set(attributes, EDXParseKeyValue(body, "=", "|", false, "last"))
In this example the last value is kept and the rest are dropped.
{
"attributes": {
"key1": "three"
},
"body": "key1=1.1|key1=2|key1=three"
}
Append Values
set(attributes, EDXParseKeyValue(body, "=", "|", false, "append"))
In this example the values are all included as an array.
{
"attributes": {
"key1": [
"1.1",
"2",
"three"
]
},
"body": "key1=1.1|key1=2|key1=three"
}
Concatenate Values
set(attributes, EDXParseKeyValue(body, "=", "|", false, "concat"))
In this example the values are all included as a single comma separated value.
{
"attributes": {
"key1": "1.1, 2, three"
},
"body": "key1=1.1|key1=2|key1=three"
}
Index Values
set(attributes, EDXParseKeyValue(body, "=", "|", false, "indexed"))
In this example the values are all included by creating multiple indexed keys.
{
"attributes": {
"key1_0": "1.1",
"key1_1": "2",
"key1_2": "three"
},
"body": "key1=1.1|key1=2|key1=three"
}
Convert strings to int or float
set(attributes, EDXParseKeyValue(body, "=", "|", true, "indexed"))
In this example some strings are detected as being an integer and a float and converted into the appropriate data types. The string remains unchanged.
{
"attributes": {
"key1_0": 1.1,
"key1_1": 2,
"key1_2": "three"
},
"body": "key1=1.1|key1=2|key1=three"
}
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: ottl_transform
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
statements
The statements
parameter defines a collection of OTTL expressions that specify how each data item should be transformed. Each statement is written using OTTL syntax as a block scalar in YAML format, allowing multiple transformation instructions to be included within a single statements parameter. The pipe character |
signifies that the subsequent content is a multi-line string.
nodes:
- name: <name>
type: ottl_transform
statements: |
<OTTL expression>
<OTTL expression>
<OTTL expression>
Note: When defining multiple transformation statements, ensure that each statement appropriately considers data dependencies and transformation order.