Interpret Edge Delta Data Items

Overview

As of agent version 0.1.78, incoming logs are handled using the OTEL schema. All fields of the incoming log become the body field, and OTEL parameters such as resource are generated by the agent.

The log body is formatted as a JSON string including the addition of escaping characters.

Consider an error message: The error code "ERR\57" occurred in Module A..

When ingested by the agent and escaped into JSON, it appears in the pipeline as follows:

{
  "body": "The error code \"ERR\\57\" occurred in Module A."
}

Note how special characters are escaped.

This is relevant when configuring agent nodes that contain CEL and when referencing any field path. See Regex as a String.

Note: If you ingest logs using the OTEL source node, the existing OTEL parameters such as attributes and resources are mapped natively into the data item rather than packaged into the body field.

OTEL Parameters

Field References

Use bracket notation to reference fields. Consider the following log:

{
  "_type": "log"
  "attributes": {
    "newfield": "I added this new field value"
  }
  "body": "{"timestamp":"2023-04-23T12:34:56.789Z","logLevel":"ERROR","serviceName":"AuthService","nodeId":"node2","message":"Login failed","clientIP":"192.168.1.10","username":"user123","event":"login_attempt","outcome":"failure"}"
  "resource": {
    "config_id": "87654321-1321-69874-9456-s5123456h7"
    "ed.tag": "ed_parallel"
    "host.name": "ED_TEST"
    "ip": "10.0.0.1"
    "src_type": ""
  }
  "timestamp": 1703677287365
}

The following references can be used to specify fields in this log:

• item["body"]
• item["attributes"]["newfield"]
• item["resource"]["host.name"]

Tip: A CEL macro can be used to reference fields in a JSON structured body: json(item["body"]).event

The resources section references source components when using Kubernetes. Like host.name, it uses periods that do not indicate nesting. This means that when referencing resources in the UI form or in a YAML file you should use the following format:

item.resource["k8s.namespace.name"]

Bear in mind, some Edge Delta configuration fields use dot notation, such as the field_path in a log transform node. Refer to the documentation for each node for details about how to specify each parameter.

Internal Edge Delta Fields

The Edge Delta agent name is the one specified when you create the Fleet.

• item["resource"]["ed.tag"]

This field shows the ID for the organization the Edge Delta agent was created in.

• item["resource"]["ed.org.id"]

This field shows the API key for the pipeline the Edge Delta agent is configured with.

• item["resource"]["config_id"]

Fields starting with a double underscore are added to the data items for internal use within the agent. These are stripped out before being sent to Loki, Prometheus or GCL destinations. If you require them you need to add them as labels.

• item["resource"]["__group_name"]
• item["resource"]["__logical_source"]
• item["resource"]["__short_src_name"]
• item["resource"]["__src_name"]

Edge Delta Agent Logs

The Edge Delta agents generate their own logs using a service called ed-agent-log. These logs are not included when calculating usage volumes for your selected plan.

Data Item Splitting

As of v1.13.0, the agent will split any incoming message larger than 1Mb into individual messages. No telemetry data will be lost, it will be split into multiple, smaller messages. Each resulting message will become a completely independent telemetry message - each split message will carry full metadata with the split log body.

In addition, each split message will contain the following new attributes:

• ed.split.uid: a UUID for the message
• ed.split.index: the 0-based index indicating the position of this message within the sequence of messages that resulted from the split.
• ed.split.total_count: the total number of messages that resulted from the split

When pushing post-processed telemetry data to a downstream destination like S3 or GCL, this change will result in each split message being pushed to the downstream destination as a separate message. However, you can reassemble the messages using the new attributes. As a best practice, it is not advisable to have individual log or telemetry messages larger than 1Mb.