Interpret Edge Delta Data Items

Data items are handled by nodes and transported by links.

  8 minute read  

Overview

As of agent version 0.1.78, incoming logs are handled using the OTEL schema. All fields of the incoming log become the body field, and OTEL parameters such as resource are generated by the agent.

The log body is formatted as a JSON string including the addition of escaping characters.

Consider an error message: The error code "ERR\57" occurred in Module A..

When ingested by the agent and escaped into JSON, it appears in the pipeline as follows:

{
  "body": "The error code \"ERR\\57\" occurred in Module A."
}

Note how special characters are escaped.

This is relevant when configuring agent nodes that contain CEL and when referencing any field path. See Regex as a String.

Note: If you ingest logs using the OTEL source node, the existing OTEL parameters such as attributes and resources are mapped natively into the data item rather than packaged into the body field.

OTEL Parameters

Field Name Description
timestamp Time when the event occurred as measured by the source. It is formatted as uint64 nanoseconds since Unix epoch.
severity_text A description of the log level.
body The body of the log record.
resource The resource parameter uses a map to list the resources and tags that describe the source of the log.
_type The OTEL signal type.
attributes Event specific information such as the resource, custom field or other context.

Field References

Use bracket notation to reference fields. Consider the following log:


{
  "_type": "log"
  "attributes": {
    "newfield": "I added this new field value"
  }
  "body": "{"timestamp":"2023-04-23T12:34:56.789Z","logLevel":"ERROR","serviceName":"AuthService","nodeId":"node2","message":"Login failed","clientIP":"192.168.1.10","username":"user123","event":"login_attempt","outcome":"failure"}"
  "resource": {
    "config_id": "87654321-1321-69874-9456-s5123456h7"
    "ed.tag": "ed_parallel"
    "host.name": "ED_TEST"
    "ip": "10.0.0.1"
    "src_type": ""
  }
  "timestamp": 1703677287365
}

The following references can be used to specify fields in this log:

  • item["body"]
  • item["attributes"]["newfield"]
  • item["resource"]["host.name"]

Tip: A CEL macro can be used to reference fields in a JSON structured body: json(item["body"]).event

The resources section references source components when using Kubernetes. Like host.name, it uses periods that do not indicate nesting. This means that when referencing resources in the UI form or in a YAML file you should use the following format:

item.resource["k8s.namespace.name"]

Bear in mind, some Edge Delta configuration fields use dot notation, such as the field_path in a log transform node. Refer to the documentation for each node for details about how to specify each parameter.

Internal Edge Delta Fields

The Edge Delta agent name is the one specified when you create the Fleet.

  • item["resource"]["ed.tag"]

This field shows the ID for the organization the Edge Delta agent was created in.

  • item["resource"]["ed.org.id"]

This field shows the API key for the pipeline the Edge Delta agent is configured with.

  • item["resource"]["config_id"]

Fields starting with a double underscore are added to the data items for internal use within the agent. These are stripped out before being sent to Loki, Prometheus or GCL destinations. If you require them you need to add them as labels.

  • item["resource"]["__group_name"]
  • item["resource"]["__logical_source"]
  • item["resource"]["__short_src_name"]
  • item["resource"]["__src_name"]

Data Item Splitting

As of v1.13.0, the agent will split any incoming message larger than 1Mb into individual messages. No telemetry data will be lost, it will be split into multiple, smaller messages. Each resulting message will become a completely independent telemetry message - each split message will carry full metadata with the split log body.

In addition, each split message will contain the following new attributes:

  • ed.split.uid: a UUID for the message
  • ed.split.index: the 0-based index indicating the position of this message within the sequence of messages that resulted from the split.
  • ed.split.total_count: the total number of messages that resulted from the split

When pushing post-processed telemetry data to a downstream destination like S3 or GCL, this change will result in each split message being pushed to the downstream destination as a separate message. However, you can reassemble the messages using the new attributes. As a best practice, it is not advisable to have individual log or telemetry messages larger than 1Mb.

Example Kubernetes Source

The following example is a log for a Kubernetes source after passing the source node and in this example an enrichment node to add attributes.

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
  },
  "_type": "log",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "user": "userA",
    "operation": "delete",
    "outcome": "failed",
    "k8s_labels": {
      "app":"my-api"
    },
    "k8s_annotations": {
      "service": "edgedelta",
    },
  },  
}

Example Docker Source

The following example illustrates the OTEL schema for a Docker source.

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "container.name": "container_name",
    "container.image.name": "container_image_name",
  },
  "attributes": {
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "user": "userA",
    "operation": "delete",
    "outcome": "failed",
  },  
}

Example File Source

The following example illustrates the OTEL schema for a File input.

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "ed.filepath": "filepath"
  },
  "attributes": {
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "user": "userA",
    "operation": "delete",
    "outcome": "failed",
  },  
}

Example Cluster Pattern and Sample

The following example illustrates the cluster pattern item.

{
  "_additional_samples": [
    "0": "2024-08-05 01:35:58 - frauddetectionservice - Consumed record with orderId: 108eed2b-52cb-11ef-b57c-fa18d89da6c7, and updated total count to: 19 trace_id=66a461d0ce3d568544f08515caa26fdb span_id=487a3b23459fcd1a trace_flags=01 "
    "1": "2024-08-05 01:36:13 - frauddetectionservice - Consumed record with orderId: 19a4ff10-52cb-11ef-b57c-fa18d89da6c7, and updated total count to: 20 trace_id=b2598c0f08d6299a8c99ba8aaa3e2bc9 span_id=f3108b89dc2e881c trace_flags=01 "
    "length": 2
  ]
  "_pattern": "* frauddetectionservice Consumed record with orderId * and updated total count to *"
  "_pattern_count": 3
  "_sample": "2024-08-05 01:35:44 - frauddetectionservice - Consumed record with orderId: 08245d06-52cb-11ef-b57c-fa18d89da6c7, and updated total count to: 18 trace_id=70d4f6dd654bb97bd2242c72939303eb span_id=24275a437206107e trace_flags=01 "
  "_sentiment_score": 0
  "_type": "cluster_pattern_and_sample"
  "resource": { • • • }
  "start_timestamp": 1722821751522
  "timestamp": 1722821781522
}

Example Metric Count Item

The following example illustrates the OTEL schema for a metric count item.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
  },
  "_stat_type": "count",
  "name": "error.count",
  "kind": "sum",
  "sum": {
    "is_monotonic": false,
    "value": 10
  }
}

Example Metric Sum Item

The following example illustrates the OTEL schema for a metric sum item.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
  },  
  "_stat_type": "sum",
  "name": "request_count.sum",
  "kind": "sum",
  "sum": {
    "is_monotonic": true,
    "value": 10000
  }
}

Example Generic Metric Item

The following example illustrates the OTEL schema for a metric item other than count and sum.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
  },  
  "_stat_type": "anomaly1",
  "name": "error.anomaly1",
  "kind": "gauge",
  "gauge": {
    "value": 99
  }
}

Example Event Item

The following example is an event item:

{
  "_type": "log"
  "attributes": {
    "event.count": "1"
    "event.firstTimestamp": "2024-07-31T11:39:38Z"
    "event.lastTimestamp": "2024-07-31T11:39:38Z"
    "event.metadata.creationTimestamp": "2024-07-31T11:39:38Z"
    "event.metadata.namespace": "demo"
    "event.metadata.resourceVersion": "1903"
    "event.metadata.uid": "6d2ba31c-2962-49b2-b058-a37424f2c5d7"
    "event.reason": "Pulled"
    "event.source.component": "kubelet"
    "event.source.host": "parallelcluster-worker2"
    "event.type": "Normal"
    "item.type": "event"
  }
  "body": "Successfully pulled image "ghcr.io/demo/demo:1.11.1-kafka" in 1m6.221188531s (1m27.422871666s including waiting)"
  "resource": {
    "container.id": ""
    "container.image.name": "ghcr.io/demo/demo:1.11.1-kafka"
    "ed.conf.id": "123456789987654321"
    "ed.filepath": ""
    "ed.org.id": "987654321123456789"
    "ed.tag": "parallelimage"
    "event.domain": "K8s"
    "event.name": "[Normal] Successfully pulled image "ghcr.io/demo/demo:1.11.1-kafka" in 1m6.221188531s (1m27.422871666s including waiting)"
    "host.ip": "172.18.0.2"
    "host.name": "parallelcluster-worker2"
    "k8s.container.name": "kafka"
    "k8s.deployment.name": "demo-kafka"
    "k8s.namespace.name": "demo"
    "k8s.node.name": "parallelcluster-worker2"
    "k8s.pod.name": "demo-kafka-6d5d7474db-pb8df"
    "k8s.pod.uid": "d1159af9-493f-4d7c-a675-5c5b1f86ac30"
    "k8s.replicaset.name": "demo-kafka-6d5d7474db"
    "service.name": ""
    "src_type": "K8s"
  }
  "severity_text": "info"
  "timestamp": 1722425978000
}

Example Signal

The following example is a signal item:

[
	{
		"_type": "signal",
		"resource": {
			"ed.conf.id": "12345678-1x234-4abc-def5-12345678910g",
			"ed.org.id": "1098765a-432b-1cde-2345-fg6789hij101",
			"ed.tag": "parallel_container",
			"host.ip": "10.0.0.1",
			"host.name": "ED_TEST",
			"src_type": "memory_input"
		},
		"signal": {
			"description": "delivery_time_by_platform_delivery_time.avg hit threshold threshold_test-threshold-checker of filter: item.name == \"delivery_time_by_platform_delivery_time.avg\" and condition: value > 40 with value 40.67",
			"name": "delivery_time_by_platform_delivery_time.avg",
			"signal_id": "850666",
			"threshold_condition": "value > 40",
			"threshold_filter": "item.name == \"delivery_time_by_platform_delivery_time.avg\"",
			"title": "Threshold threshold_test-threshold-checker triggered",
			"value": 40.666666666666664
		},
		"timestamp": 1715683146563
	}
]

Example Health Item

{
  "_health": {
    "component": "mask_filter"
    "name": "my-mask-processor"
    "properties": {
      "last_error": "<nil>"
      "mask_test-mask-processor.error.count.one_minute": 0
      "mask_test-mask-processor.hit.count.one_hour": 0
      "mask_test-mask-processor.last_ten_min_err_count.count.ten_minutes": 0
    }
    "running": 
true
    "status": "ok"
    "type": "agentComponentHealth"
  }
  "_type": "health"
  "resource": {
    "ed.conf.id": "12345678987654321"
    "ed.org.id": "98765432123456789"
    "ed.tag": "my-cluster"
    "host.ip": ""
    "host.name": "mycluster-worker"
  }
  "timestamp": 1723431526000
}

Example Trace


{
  "_type": "trace"
  "attributes": {
    "component": "proxy"
    "ed.event.subtype": "HTTP"
    "ed.event.type": "HTTP"
    "ed.span.resource": "router frontend egress"
    "ed.status_code": "200"
    "http.protocol": "HTTP/1.1"
    "http.status_code": "200"
    "otel.scope.name": ""
    "otel.scope.version": ""
    "peer.address": "10.96.72.33:8080"
    "response_flags": "-"
    "upstream_address": "10.96.72.33:8080"
    "upstream_cluster": "frontend"
    "upstream_cluster.name": "frontend"
  }
  "end_time_unix_nano": 1727694533408816000
  "events": 
null
  "links": 
null
  "parent.span.id": "ffd797d422da853a"
  "resource": {
    "ed.conf.id": "123456789"
    "ed.org.id": "987654321"
    "ed.tag": "OTEL"
    "host.ip": "172.18.0.4"
    "host.name": "otel-cluster-worker"
    "k8s.deployment.name": "my-otel-demo-frontendproxy"
    "k8s.namespace.name": "otel"
    "k8s.node.name": "otel-cluster-worker"
    "k8s.pod.ip": "10.244.1.12"
    "k8s.pod.name": "my-otel-demo-frontendproxy-6c4d5f5d4d-lbbtk"
    "k8s.pod.start_time": "2024-09-28T02:28:04Z"
    "k8s.pod.uid": "afac6ac0-9458-493a-a2d5-d8be273d4cfc"
    "server.port": 4326
    "service.instance.id": "afac6ac0-9458-493a-a2d5-d8be273d4cfc"
    "service.name": "frontendproxy"
    "service.namespace": "opentelemetry-demo"
    "service.version": "1.11.1"
    "src_type": "otlp"
  }
  "span.duration": 181499000
  "span.id": "22b53dad5e808ec6"
  "span.kind": "SPAN_KIND_CLIENT"
  "span.name": "router frontend egress"
  "start_time_unix_nano": 1727694533227317000
  "status.code": "STATUS_CODE_OK"
  "status.message": ""
  "timestamp": 1727694533227
  "trace.id": "4c27c6e851cf08878b87907f9edd600e"
  "trace.state": ""
}

Custom Data Item

A custom data item refers to a data item with a schema that does not align with a common structure. These data items may be useful for bespoke applications or configurations, but they may cause validation errors.