Edge Delta Data Item

Data items are handled by nodes and transported by links.

  6 minute read  

Paid
Visual Pipelines Beta

Overview

As of agent version 0.1.78, incoming logs are handled using the OTEL schema. The incoming log becomes the body field and OTEL parameters such as resource are added by the input nodes. This is relevant when configuring agent nodes that contain CEL and when referencing any field path.

Note: If you ingest logs using the OTEL input the OTEL parameters such as attributes and resources are mapped natively into the data item rather than packaged into the body field.

OTEL Parameters

Field Name Description
Timestamp Time when the event occurred as measured by the source. It is formatted as uint64 nanoseconds since Unix epoch.
severity_text A description of the log level.
body The body of the log record.
resource The resource parameter uses a map to list the resources and tags that describe the source of the log.
_type The OTEL signal type.
attributes Event specific information such as the resource, custom field or other context.

Field References

Use bracket notation to reference fields. Consider the following log:


{
  "_type": "log"
  "attributes": {
    "newfield": "I added this new field value"
  }
  "body": "{"timestamp":"2023-04-23T12:34:56.789Z","logLevel":"ERROR","serviceName":"AuthService","nodeId":"node2","message":"Login failed","clientIP":"192.168.1.10","username":"user123","event":"login_attempt","outcome":"failure"}"
  "resource": {
    "config_id": "87654321-1321-69874-9456-s5123456h7"
    "ed.tag": "ed_parallel"
    "host.name": "ED_TEST"
    "ip": "10.0.0.1"
    "src_type": ""
  }
  "timestamp": 1703677287365
}

The following references can be used to specify fields in this log:

  • item["body"]
  • item["attributes"]["newfield"]
  • item["resource"]["host.name"]

Tip: A CEL macro can be used to reference fields in a JSON structured body: json(item["body"]).event

You can copy a log path in the bracket notation using the processor test pane. This helps prevent validation failure in agent configurations. In the pipeline view:

  1. Click Edit Mode
  2. Select a node
  3. Click Test
  4. Open the Processor tab
  5. Select Test in Isolation
  6. Paste your log input
  7. Click the field in the output and select Copy Path to clipboard.

Example Kubernetes Input

The following example is a log for a Kubernetes input after passing the input node and in this example an enrichment node to add attributes.

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
  },
  "_type": "log",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "user": "userA",
    "operation": "delete",
    "outcome": "failed",
    "k8s_labels": {
      "app":"my-api"
    },
    "k8s_annotations": {
      "service": "edgedelta",
    },
  },  
}

timestamp

The time at which the log was created. The timestamp is created by the source component.

body

The body of the log containing the raw data.

resource

host.name This is a semantic convention from OTEL for the host server name or IP address.

ed.tag The Edge Delta agent name specified when you create the agent configuration.

ed.org.id The ID for the organization the Edge Delta agent was created in.

__parameter Parameters starting with a double underscore are internal for use by the Edge Delta backend.

Source Components The resources section also references source components. It uses periods that do not indicate nesting. This means that when referencing resources in the UI form or in a YAML file you should use the following format:

item.resource["k8s.namespace.name"]

_type

The OTEL signal type.

attributes

Custom fields such as user enrichments and parsed fields are contained in the attributes section.

Example Docker Input

The following example illustrates the OTEL schema for a Docker input.

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "container.name": "container_name",
    "container.image.name": "container_image_name",
  },
  "attributes": {
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "user": "userA",
    "operation": "delete",
    "outcome": "failed",
  },  
}

Example File Input

The following example illustrates the OTEL schema for a File input.

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": "{\"user\": \"userA\",\"operation\": \"delete\",\"outcome\": \"failed\"}",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "ed.filepath": "filepath"
  },
  "attributes": {
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "user": "userA",
    "operation": "delete",
    "outcome": "failed",
  },  
}

Example Cluster Pattern Signal

The following example illustrates the OTEL schema for a cluster pattern signal.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
  },
  "attributes": {
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
    "role": "admin",  // all patterns have this value
    "service": "db-service", // all patterns have this value
    "ip": "*" //multiple value 
  },  
  "_type": "cluster_pattern_and_sample",
  "_pattern": "* GET * HTTP* user agent* nginx *",
  "_pattern_count": 2,
  "_sample": "[10.212.85.54] \"GET /api/v1/test?foo=bar HTTP/1.1\" 200 214 \"-\" \"user-agent/10.229.21.0\" 0.010 0.010 \"127.0.0.1:10001\" \"10.225.202.219\" [nginx] \"10.229.21.0\"",
  "_additional_samples": [
      "[10.222.90.55] \"GET /api/v2/test?foo=bar HTTP/1.1\" 500 250 \"-\" \"user-agent/10.222.90.55\" 0.000 0.000 \"127.0.0.1:10002\" \"10.222.90.55\" [nginx] \"10.222.90.55\""
  ],
  "_sentiment_score": 0.00
}

Example Metric Count Item

The following example illustrates the OTEL schema for a metric count item.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
  },
  "_stat_type": "count",
  "name": "error.count",
  "kind": "sum",
  "sum": {
    "is_monotonic": false,
    "value": 10
  }
}

Example Metric Sum Item

The following example illustrates the OTEL schema for a metric sum item.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
  },  
  "_stat_type": "sum",
  "name": "request_count.sum",
  "kind": "sum",
  "sum": {
    "is_monotonic": true,
    "value": 10000
  }
}

Example Generic Metric Item

The following example illustrates the OTEL schema for a metric item other than count and sum.

{
  "timestamp": "1581452773000000789",
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "config_id": "12345678-abcd-...",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
    "k8s.labels.app": "my-api",
  },
  "_type": "metric",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s.pod.labels": {
      "app":"my-api"
    },
    "k8s.pod.annotation": {
      "service": "edgedelta",
    },
  },  
  "_stat_type": "anomaly1",
  "name": "error.anomaly1",
  "kind": "gauge",
  "gauge": {
    "value": 99
  }
}