Interpret Edge Delta Data Items
8 minute read
Overview
As of agent version 0.1.78, incoming logs are handled using the OTEL schema. All fields of the incoming log become the body field, and OTEL parameters such as resource
are generated by the agent. This is relevant when configuring agent nodes that contain CEL and when referencing any field path. See Regex as a String.
Note: If you ingest logs using the OTEL source node, the existing OTEL parameters such as attributes and resources are mapped natively into the data item rather than packaged into the body field.
OTEL Parameters
Field Name | Description |
---|---|
timestamp |
Time when the event occurred as measured by the source. It is formatted as uint64 nanoseconds since Unix epoch. |
severity_text |
A description of the log level. |
body |
The body of the log record. |
resource |
The resource parameter uses a map to list the resources and tags that describe the source of the log. |
_type |
The OTEL signal type. |
attributes |
Event specific information such as the resource, custom field or other context. |
Field References
Use bracket notation to reference fields. Consider the following log:
{
"_type": "log"
"attributes": {
"newfield": "I added this new field value"
}
"body": "{"timestamp":"2023-04-23T12:34:56.789Z","logLevel":"ERROR","serviceName":"AuthService","nodeId":"node2","message":"Login failed","clientIP":"192.168.1.10","username":"user123","event":"login_attempt","outcome":"failure"}"
"resource": {
"config_id": "87654321-1321-69874-9456-s5123456h7"
"ed.tag": "ed_parallel"
"host.name": "ED_TEST"
"ip": "10.0.0.1"
"src_type": ""
}
"timestamp": 1703677287365
}
The following references can be used to specify fields in this log:
item["body"]
item["attributes"]["newfield"]
item["resource"]["host.name"]
Tip: A CEL macro can be used to reference fields in a JSON structured body:
json(item["body"]).event
The resources section references source components when using Kubernetes. Like host.name
, it uses periods that do not indicate nesting. This means that when referencing resources in the UI form or in a YAML file you should use the following format:
item.resource["k8s.namespace.name"]
Internal Edge Delta Fields
The Edge Delta agent name is the one specified when you create the Fleet.
item["resource"]["ed.tag"]
This field shows the ID for the organization the Edge Delta agent was created in.
item["resource"]["ed.org.id"]
This field shows the API key for the pipeline the Edge Delta agent is configured with.
item["resource"]["config_id"]
Fields starting with a double underscore are added to the data items for internal use within the agent. These are stripped out before being sent to Loki, Prometheus or GCL destinations. If you require them you need to add them as labels.
item["resource"]["__group_name"]
item["resource"]["__logical_source"]
item["resource"]["__short_src_name"]
item["resource"]["__src_name"]
Data Item Splitting
As of v1.13.0, the agent will split any incoming message larger than 1Mb into individual messages. No telemetry data will be lost, it will be split into multiple, smaller messages. Each resulting message will become a completely independent telemetry message - each split message will carry full metadata with the split log body.
In addition, each split message will contain the following new attributes:
ed.split.uid
: a UUID for the messageed.split.index
: the 0-based index indicating the position of this message within the sequence of messages that resulted from the split.ed.split.total_count
: the total number of messages that resulted from the split
When pushing post-processed telemetry data to a downstream destination like S3 or GCL, this change will result in each split message being pushed to the downstream destination as a separate message. However, you can reassemble the messages using the new attributes. As a best practice, it is not advisable to have individual log or telemetry messages larger than 1Mb.
Example Kubernetes Source
The following example is a log for a Kubernetes source after passing the source node and in this example an enrichment node to add attributes.
{
"timestamp": "1581452773000000789",
"severity_text": "Error",
"body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
},
"_type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"user": "userA",
"operation": "delete",
"outcome": "failed",
"k8s_labels": {
"app":"my-api"
},
"k8s_annotations": {
"service": "edgedelta",
},
},
}
Example Docker Source
The following example illustrates the OTEL schema for a Docker source.
{
"timestamp": "1581452773000000789",
"severity_text": "Error",
"body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"container.name": "container_name",
"container.image.name": "container_image_name",
},
"attributes": {
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"user": "userA",
"operation": "delete",
"outcome": "failed",
},
}
Example File Source
The following example illustrates the OTEL schema for a File input.
{
"timestamp": "1581452773000000789",
"severity_text": "Error",
"body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"ed.filepath": "filepath"
},
"attributes": {
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"user": "userA",
"operation": "delete",
"outcome": "failed",
},
}
Example Cluster Pattern and Sample
The following example illustrates the cluster pattern item.
{
"_additional_samples": [
"0": "2024-08-05 01:35:58 - frauddetectionservice - Consumed record with orderId: 108eed2b-52cb-11ef-b57c-fa18d89da6c7, and updated total count to: 19 trace_id=66a461d0ce3d568544f08515caa26fdb span_id=487a3b23459fcd1a trace_flags=01 "
"1": "2024-08-05 01:36:13 - frauddetectionservice - Consumed record with orderId: 19a4ff10-52cb-11ef-b57c-fa18d89da6c7, and updated total count to: 20 trace_id=b2598c0f08d6299a8c99ba8aaa3e2bc9 span_id=f3108b89dc2e881c trace_flags=01 "
"length": 2
]
"_pattern": "* frauddetectionservice Consumed record with orderId * and updated total count to *"
"_pattern_count": 3
"_sample": "2024-08-05 01:35:44 - frauddetectionservice - Consumed record with orderId: 08245d06-52cb-11ef-b57c-fa18d89da6c7, and updated total count to: 18 trace_id=70d4f6dd654bb97bd2242c72939303eb span_id=24275a437206107e trace_flags=01 "
"_sentiment_score": 0
"_type": "cluster_pattern_and_sample"
"resource": { • • • }
"start_timestamp": 1722821751522
"timestamp": 1722821781522
}
Example Metric Count Item
The following example illustrates the OTEL schema for a metric count item.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.labels.app": "my-api",
},
"_type": "metric",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
},
"_stat_type": "count",
"name": "error.count",
"kind": "sum",
"sum": {
"is_monotonic": false,
"value": 10
}
}
Example Metric Sum Item
The following example illustrates the OTEL schema for a metric sum item.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.labels.app": "my-api",
},
"_type": "metric",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
},
"_stat_type": "sum",
"name": "request_count.sum",
"kind": "sum",
"sum": {
"is_monotonic": true,
"value": 10000
}
}
Example Generic Metric Item
The following example illustrates the OTEL schema for a metric item other than count and sum.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.labels.app": "my-api",
},
"_type": "metric",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
},
"_stat_type": "anomaly1",
"name": "error.anomaly1",
"kind": "gauge",
"gauge": {
"value": 99
}
}
Example Event Item
The following example is an event item:
{
"_type": "log"
"attributes": {
"event.count": "1"
"event.firstTimestamp": "2024-07-31T11:39:38Z"
"event.lastTimestamp": "2024-07-31T11:39:38Z"
"event.metadata.creationTimestamp": "2024-07-31T11:39:38Z"
"event.metadata.namespace": "demo"
"event.metadata.resourceVersion": "1903"
"event.metadata.uid": "6d2ba31c-2962-49b2-b058-a37424f2c5d7"
"event.reason": "Pulled"
"event.source.component": "kubelet"
"event.source.host": "parallelcluster-worker2"
"event.type": "Normal"
"item.type": "event"
}
"body": "Successfully pulled image "ghcr.io/demo/demo:1.11.1-kafka" in 1m6.221188531s (1m27.422871666s including waiting)"
"resource": {
"container.id": ""
"container.image.name": "ghcr.io/demo/demo:1.11.1-kafka"
"ed.conf.id": "123456789987654321"
"ed.filepath": ""
"ed.org.id": "987654321123456789"
"ed.tag": "parallelimage"
"event.domain": "K8s"
"event.name": "[Normal] Successfully pulled image "ghcr.io/demo/demo:1.11.1-kafka" in 1m6.221188531s (1m27.422871666s including waiting)"
"host.ip": "172.18.0.2"
"host.name": "parallelcluster-worker2"
"k8s.container.name": "kafka"
"k8s.deployment.name": "demo-kafka"
"k8s.namespace.name": "demo"
"k8s.node.name": "parallelcluster-worker2"
"k8s.pod.name": "demo-kafka-6d5d7474db-pb8df"
"k8s.pod.uid": "d1159af9-493f-4d7c-a675-5c5b1f86ac30"
"k8s.replicaset.name": "demo-kafka-6d5d7474db"
"service.name": ""
"src_type": "K8s"
}
"severity_text": "info"
"timestamp": 1722425978000
}
Example Signal
The following example is a signal item:
[
{
"_type": "signal",
"resource": {
"ed.conf.id": "12345678-1x234-4abc-def5-12345678910g",
"ed.org.id": "1098765a-432b-1cde-2345-fg6789hij101",
"ed.tag": "parallel_container",
"host.ip": "10.0.0.1",
"host.name": "ED_TEST",
"src_type": "memory_input"
},
"signal": {
"description": "delivery_time_by_platform_delivery_time.avg hit threshold threshold_test-threshold-checker of filter: item.name == \"delivery_time_by_platform_delivery_time.avg\" and condition: value > 40 with value 40.67",
"name": "delivery_time_by_platform_delivery_time.avg",
"signal_id": "850666",
"threshold_condition": "value > 40",
"threshold_filter": "item.name == \"delivery_time_by_platform_delivery_time.avg\"",
"title": "Threshold threshold_test-threshold-checker triggered",
"value": 40.666666666666664
},
"timestamp": 1715683146563
}
]
Example Health Item
{
"_health": {
"component": "mask_filter"
"name": "my-mask-processor"
"properties": {
"last_error": "<nil>"
"mask_test-mask-processor.error.count.one_minute": 0
"mask_test-mask-processor.hit.count.one_hour": 0
"mask_test-mask-processor.last_ten_min_err_count.count.ten_minutes": 0
}
"running":
true
"status": "ok"
"type": "agentComponentHealth"
}
"_type": "health"
"resource": {
"ed.conf.id": "12345678987654321"
"ed.org.id": "98765432123456789"
"ed.tag": "my-cluster"
"host.ip": ""
"host.name": "mycluster-worker"
}
"timestamp": 1723431526000
}
Example Trace
{
"_type": "trace"
"attributes": {
"component": "proxy"
"ed.event.subtype": "HTTP"
"ed.event.type": "HTTP"
"ed.span.resource": "router frontend egress"
"ed.status_code": "200"
"http.protocol": "HTTP/1.1"
"http.status_code": "200"
"otel.scope.name": ""
"otel.scope.version": ""
"peer.address": "10.96.72.33:8080"
"response_flags": "-"
"upstream_address": "10.96.72.33:8080"
"upstream_cluster": "frontend"
"upstream_cluster.name": "frontend"
}
"end_time_unix_nano": 1727694533408816000
"events":
null
"links":
null
"parent.span.id": "ffd797d422da853a"
"resource": {
"ed.conf.id": "123456789"
"ed.org.id": "987654321"
"ed.tag": "OTEL"
"host.ip": "172.18.0.4"
"host.name": "otel-cluster-worker"
"k8s.deployment.name": "my-otel-demo-frontendproxy"
"k8s.namespace.name": "otel"
"k8s.node.name": "otel-cluster-worker"
"k8s.pod.ip": "10.244.1.12"
"k8s.pod.name": "my-otel-demo-frontendproxy-6c4d5f5d4d-lbbtk"
"k8s.pod.start_time": "2024-09-28T02:28:04Z"
"k8s.pod.uid": "afac6ac0-9458-493a-a2d5-d8be273d4cfc"
"server.port": 4326
"service.instance.id": "afac6ac0-9458-493a-a2d5-d8be273d4cfc"
"service.name": "frontendproxy"
"service.namespace": "opentelemetry-demo"
"service.version": "1.11.1"
"src_type": "otlp"
}
"span.duration": 181499000
"span.id": "22b53dad5e808ec6"
"span.kind": "SPAN_KIND_CLIENT"
"span.name": "router frontend egress"
"start_time_unix_nano": 1727694533227317000
"status.code": "STATUS_CODE_OK"
"status.message": ""
"timestamp": 1727694533227
"trace.id": "4c27c6e851cf08878b87907f9edd600e"
"trace.state": ""
}
Custom Data Item
A custom data item refers to a data item with a schema that does not align with a common structure. These data items may be useful for bespoke applications or configurations, but they may cause validation errors.