Edge Delta Datadog Mapper

Create a payload for the Datadog event endpoint.

5 minute read

Overview

The Datadog Mapper node applies mappings to create a new payload for the Datadog event endpoint. The output of this node can only be received by the Datadog destination node. This node can ingest a log type or a custom type. There are some restrictions around what types of values can populate certain fields, for example the “message” field of the Datadog payload can only take string values, JSON strings, and JSON/maps.

incoming_data_types: custom, log, cluster_pattern_and_sample
outgoing_data_types: datadog_payload

Most of the fields in the schema can be customized, except “event” and “index”. If not customized, it will use the default value:

Message: item["body"]
Tags: item["attributes"]
Source: item["resource"]["src_type"]
Source Name: item["resource"]["__src_name"]
Host: item["resource"]["host.name"]
Service:item["resource"]["__logical_source"]
Level: INFO
DataType: Log

If the specified field is missing from the incoming item, an empty string will be used as default value.

A timestamp is not defined because the default Datadog timestamp is used. You can select a field from the payload to use in Datadog using a remapper.

Example Configuration

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body"]
    dd_service: "edgedelta"
    dd_tags: item["attributes"]

Example Input log

{
    "timestamp": "1581452773000000789",
    "body": "Failed to do something",
    "resource": {
      "host.name": "host-1",
      "ed.tag": "test-config",
      "ed.org.id": "0481a213-....",
      "__src_name": "source_name",
      "src_type": "K8s",
      "__logical_source": "logical_source",
      "__short_src_name": "short_source_name",
      "__group_name": "group_name",
      "k8s.namespace.name": "edgedelta",
      "k8s.pod.name": "api-deployment-d79fab72249c",
      "k8s.container.name": "echo:latest",
      "k8s.labels.app": "my-api",
    },
    "type": "log",
    "attributes": {
        "pod_id":"api-deployment-d79fab72249c-vtq9x",
        "instance_id":"i-1234567890abcdef0",
        "instance_name":"test-name",
        "parsed": {
          "timestamp": "2023/07/11 09:40:21",
          "msg": "Failed to do something",
        },
        "original_timestamp": "2023/07/11 09:40:21",
    }
}

Example Output

{
  "message": "Failed to do something",
  "host": "host-1",
  "sourcename": "source_name",
  "ddsource": "K8s",
  "service": "edgedelta",
  "level": "INFO",
  "datatype": "Log",
  "ddtags": {
        "pod_id":"api-deployment-d79fab72249c-vtq9x",
        "instance_id":"i-1234567890abcdef0",
        "instance_name":"test-name",
        "parsed": {
          "timestamp": "2023/07/11 09:40:21",
          "msg": "Failed to do something",
        },
        "original_timestamp": "2023/07/11 09:40:21",
    },
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: datadog_mapper

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

dd_message

The dd_message parameter is used to define which input item to use for the message field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]

dd_service

The dd_service parameter is used to define which input item to use for the service field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_service: item["resource"]["__logical_source"]

dd_source

The dd_source parameter is used to define which input item to use for the ddsource field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_source: item["resource"]["__src_name"]

dd_tags

The dd_tags parameter is used to define which input item to use for the “ddtags” field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_tags: item["attributes"]

Optional Parameters

dd_datatype

The dd_datatype parameter is used to define the datatype field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_datatype: Log

dd_host

The dd_host parameter is used to define which input item to use for the host field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional but host or hostname must be set. Usually hostname is not used if host is specified, however, using both can cater for more advanced Datadog configurations.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_host: item["resource"]["host.name"]

dd_hostname

The dd_hostname parameter is used to define which input item to use for the hostname field in the output Datadog payload instead of using the default agent tag. It is specified as a string and you can use CEL macros. It is optional but host or hostname must be set. Usually hostname is not used if host is specified, however, using both can cater for more advanced Datadog configurations.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_hostname: json(item["_parsed"]).hostname

dd_level

The dd_level parameter is used to define which input item to use for the level field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_level: INFO

dd_source_name

The dd_source_name parameter is used to define which input item to use for the sourcename field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional.

nodes:
  - name: datadog_payload_mapping_example
    type: datadog_mapper
    dd_message: item["body""]
    dd_source_name: item["resource"]["__src_name"]