Use OTTL in Edge Delta

OTTL Statements

Open Telemetry Transformation Language (OTTL) is a scripting language used within the OpenTelemetry ecosystem, primarily for transforming and manipulating telemetry data. It enables users to customize how telemetry data is handled, providing a way to filter, transform, and export the data. In Edge Delta, you can use an OTTL Transformation node to transform data items with OTTL, or you can use a Route node configured with OTTL to route data items with Boolean OTTL functions.

See the OTTL documentation for more information.

An OTTL statement follows a scripting structure:

<function_call>(<target_specification>,<argument_parameters>)

A function call can be an editor or a convertor.

The target specification defines the parameter that will be modified using bracket notation. In Edge Delta, root level parameters are defined with item, such as item["body"].

The argument parameters are required by the function to perform its operation, such as values to append, patterns to match, or replacement texts.

Note: To perform string functions on the body it needs to be converted from a byte array. See working with the body.

Editor Functions

Editors transform telemetry data. They are meant to modify the underlying telemetry data by applying various functions. Editors like append, delete_key and keep_keys can manipulate data maps and slices within the telemetry data. For example, append can add values to a target, converting it into an array if necessary.

append(attributes["tags"], "prod")

In this example, append is the function being called, attributes["tags"] specifies the target data structure, and "prod" is the argument to be appended to the tags attribute.

See editor functions.

Edge Delta has created bespoke OTTL extension functions.

Converter Functions

Converters provide utility functions for transforming telemetry data without altering the original data, for example, Decode, Concat, and ConvertCase.

See converter functions for a list of converters.

Converters are used within arguments to transform input values before they are applied to the target action or field. In the following example they are used to change the case within a set editor. This example sets severity_text using an attribute value, but it uses a converter to first change the value to uppercase.

Input

{
  "_type": "log",
  "body": "time=hostname=SIMPLE|product=Firewall",
  "resource": {
    "host.name": "simple-host",
    "service.name": "basic-service"
  },
  "attributes": {
    "event": {
      "level": "error"
    }
  },
  "timestamp": 1730511053177
}

Statement

set(severity_text, ConvertCase(attributes["event"]["level"], "upper"))

Output

{
  "_type": "log",
  "attributes": {
    "event": {
      "level": "error"
    }
  },
  "body": "time=hostname=SIMPLE|product=Firewall",
  "resource": {
    "host.name": "simple-host",
    "service.name": "basic-service"
  },
  "severity_text": "ERROR",
  "timestamp": 1730511053177
}

The attributes["event"]["level"] value is changed to uppercase and used as the value for severity_text.

Comments

You can include comments in OTTL statements using // characters. This is useful to document the node’s function.

// convert the event level attribute to uppercase and save it as the root severity_text. 
set(severity_text, ConvertCase(attributes["event"]["level"], "upper"))

Conditional Logic and Comparators

You can use conditional logic to execute an action only if specific conditions are met. This is often combined with comparators to evaluate expressions.

Syntax: syntax: set(target, value) where <condition> Comparators: ==, !=, >, <, >=, <=

This example sets the alert field to Critical only if the status field is error. It demonstrates conditional logic for field modification:

set(attributes["alert"], "Critical") where attributes["status"] == "error"

In this example, it converts a string representation of a duration into a duration object if the original value is a string:

set(attributes["duration"], Duration(attributes["duration_str"])) where IsString(attributes["duration_str"])

You can also use complex conditions to decide on attribute values based on multiple criteria:

set(attributes["tier"], "High Priority") where attributes["importance"] == "high" and attributes["status"] == "active"

In this case, the tier attribute is set to High Priority only if the importance is high and the status is active.

You can use comparators in conjunction with functions to create more dynamic and powerful conditions. Suppose you want to perform an action only if an integer field retry_count is greater than a threshold, say 5:

set(attributes["retry_alert"], "Too many retries") where IsInt(attributes["retry_count"]) && attributes["retry_count"] > 5

Working with the body

When a log is ingested by the Edge Delta agent, a log data item is created and the log contents are placed in the body field. Importantly, this field is a byte array. Therefore, to perform any OTTL transformation that requires the body as a string input, it needs to be decoded from byte array to a string type as part of the transformation function.

Note: You can’t directly convert the body into a string type.

Consider this log:

[
	{
		"_type": "log",
		"body": "Starting HTTP server on 10.120.15.221:8080, serving content from /usr/share/app",
		"resource": {...},
		"timestamp": 1733285404282
	}
]

To extract any IP address from the body and save it as an attribute, a set editor function is used with an embedded ExtractPatterns converter function. Embedded within the Converter function is the Decode function to convert the body type.

set(attributes, ExtractPatterns(Decode(body, "utf-8"), "(?P<ip_address>\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b)"))

Output:

[
	{
		"_type": "log",
		"attributes": {
			"ip_address": "10.120.15.221"
		},
		"body": "Starting HTTP server on 10.120.15.221:8080, serving content from /usr/share/app",
		"resource": {...},
		"timestamp": 1733285404282
	}
]

The body was converted to a string and the regex pattern identified the IP address, which was saved as an attribute using the capture name as the field.

Working with a Cache

A cache field enables you to store data temporarily while performing subsequent transformations. This field is not visible in node testing as it is meant to be temporary. In particular it is useful to create a cache version of the body that is a string type rather than a byte array. Subsequent operations within the same transformation node can operate on the cache body rather than using an embedded decode function. This may be required if the operation syntax contains a target that should point to the body, but the target field expects a key rather than a string. It is also useful for storing data types temporarily that are not allowed in the OTEL schema.

For example set(attribute, replace_pattern(target, regex, replacement)) can’t mask text from the body while setting it in an attribute because target must be a path expression. So you need an initial step of saving the decoded body in a cache, and running the replace and set function against the cached value.

Consider this log:

[
	{
		"_type": "log",
		"body": "Starting HTTP server on 10.120.15.221:8080, serving content from /usr/share/app",
		"resource": {...},
		"timestamp": 1733286563321
	}
]

set(cache["body"], Decode(body, "utf-8"))
set(attributes, ExtractPatterns(cache["body"], "(?P<ip_address>\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b)"))

[
	{
		"_type": "log",
		"attributes": {
			"ip_address": "10.120.15.221"
		},
		"body": "Starting HTTP server on 10.120.15.221:8080, serving content from /usr/share/app",
		"resource": {...},
		"timestamp": 1733286563321
	}
]

The first statement sets the cache with a decoded body. The second statement extracts the IP address from the cached version of the body without needing to decode it. Similarly, any subsequent operations that need to use the body can use the cached decoded version.

Note: The cache does not appear as part of the log metadata and it is purged when processing is complete on the data item.

See Also