Edge Delta Data Item
6 minute read
Overview
As of agent version 0.1.78, incoming logs are handled using the OTEL schema. The incoming log becomes the body field and OTEL parameters such as resource
are added by the input nodes. This is relevant when configuring agent nodes that contain CEL and when referencing any field path.
Note: If you ingest logs using the OTEL input the OTEL parameters such as attributes and resources are mapped natively into the data item rather than packaged into the body field.
OTEL Parameters
Field Name | Description |
---|---|
Timestamp | Time when the event occurred as measured by the source. It is formatted as uint64 nanoseconds since Unix epoch. |
severity_text | A description of the log level. |
body | The body of the log record. |
resource | The resource parameter uses a map to list the resources and tags that describe the source of the log. |
_type | The OTEL signal type. |
attributes | Event specific information such as the resource, custom field or other context. |
Field References
Use bracket notation to reference fields. Consider the following log:
{
"_type": "log"
"attributes": {
"newfield": "I added this new field value"
}
"body": "{"timestamp":"2023-04-23T12:34:56.789Z","logLevel":"ERROR","serviceName":"AuthService","nodeId":"node2","message":"Login failed","clientIP":"192.168.1.10","username":"user123","event":"login_attempt","outcome":"failure"}"
"resource": {
"config_id": "87654321-1321-69874-9456-s5123456h7"
"ed.tag": "ed_parallel"
"host.name": "ED_TEST"
"ip": "10.0.0.1"
"src_type": ""
}
"timestamp": 1703677287365
}
The following references can be used to specify fields in this log:
item["body"]
item["attributes"]["newfield"]
item["resource"]["host.name"]
Tip: A CEL macro can be used to reference fields in a JSON structured body:
json(item["body"]).event
You can copy a log path in the bracket notation using the processor test pane. This helps prevent validation failure in agent configurations. In the pipeline view:
- Click Edit Mode
- Select a node
- Click Test
- Open the Processor tab
- Select Test in Isolation
- Paste your log input
- Click the field in the output and select Copy Path to clipboard.
Example Kubernetes Input
The following example is a log for a Kubernetes input after passing the input node and in this example an enrichment node to add attributes.
{
"timestamp": "1581452773000000789",
"severity_text": "Error",
"body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
},
"_type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"user": "userA",
"operation": "delete",
"outcome": "failed",
"k8s_labels": {
"app":"my-api"
},
"k8s_annotations": {
"service": "edgedelta",
},
},
}
timestamp
The time at which the log was created. The timestamp is created by the source component.
body
The body of the log containing the raw data.
resource
host.name This is a semantic convention from OTEL for the host server name or IP address.
ed.tag The Edge Delta agent name specified when you create the agent configuration.
ed.org.id The ID for the organization the Edge Delta agent was created in.
__parameter Parameters starting with a double underscore are internal for use by the Edge Delta backend.
Source Components The resources section also references source components. It uses periods that do not indicate nesting. This means that when referencing resources in the UI form or in a YAML file you should use the following format:
item.resource["k8s.namespace.name"]
_type
The OTEL signal type.
attributes
Custom fields such as user enrichments and parsed fields are contained in the attributes section.
Example Docker Input
The following example illustrates the OTEL schema for a Docker input.
{
"timestamp": "1581452773000000789",
"severity_text": "Error",
"body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"container.name": "container_name",
"container.image.name": "container_image_name",
},
"attributes": {
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"user": "userA",
"operation": "delete",
"outcome": "failed",
},
}
Example File Input
The following example illustrates the OTEL schema for a File input.
{
"timestamp": "1581452773000000789",
"severity_text": "Error",
"body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"ed.filepath": "filepath"
},
"attributes": {
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"user": "userA",
"operation": "delete",
"outcome": "failed",
},
}
Example Cluster Pattern Signal
The following example illustrates the OTEL schema for a cluster pattern signal.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
},
"attributes": {
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
"role": "admin", // all patterns have this value
"service": "db-service", // all patterns have this value
"ip": "*" //multiple value
},
"_type": "cluster_pattern_and_sample",
"_pattern": "* GET * HTTP* user agent* nginx *",
"_pattern_count": 2,
"_sample": "[10.212.85.54] \"GET /api/v1/test?foo=bar HTTP/1.1\" 200 214 \"-\" \"user-agent/10.229.21.0\" 0.010 0.010 \"127.0.0.1:10001\" \"10.225.202.219\" [nginx] \"10.229.21.0\"",
"_additional_samples": [
"[10.222.90.55] \"GET /api/v2/test?foo=bar HTTP/1.1\" 500 250 \"-\" \"user-agent/10.222.90.55\" 0.000 0.000 \"127.0.0.1:10002\" \"10.222.90.55\" [nginx] \"10.222.90.55\""
],
"_sentiment_score": 0.00
}
Example Metric Count Item
The following example illustrates the OTEL schema for a metric count item.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.labels.app": "my-api",
},
"_type": "metric",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
},
"_stat_type": "count",
"name": "error.count",
"kind": "sum",
"sum": {
"is_monotonic": false,
"value": 10
}
}
Example Metric Sum Item
The following example illustrates the OTEL schema for a metric sum item.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.labels.app": "my-api",
},
"_type": "metric",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
},
"_stat_type": "sum",
"name": "request_count.sum",
"kind": "sum",
"sum": {
"is_monotonic": true,
"value": 10000
}
}
Example Generic Metric Item
The following example illustrates the OTEL schema for a metric item other than count and sum.
{
"timestamp": "1581452773000000789",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"config_id": "12345678-abcd-...",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.labels.app": "my-api",
},
"_type": "metric",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
"instance_id":"i-1234567890abcdef0", // user enrichment
"instance_name":"test-name", // user enrichment
"k8s.pod.labels": {
"app":"my-api"
},
"k8s.pod.annotation": {
"service": "edgedelta",
},
},
"_stat_type": "anomaly1",
"name": "error.anomaly1",
"kind": "gauge",
"gauge": {
"value": 99
}
}