Edge Delta Datadog Mapper
5 minute read
Overview
The Datadog Mapper node applies mappings to create a new payload for the Datadog event endpoint. The output of this node can only be received by the Datadog output node. This node can ingest a log type or a custom type. There are some restrictions around what types of values can populate certain fields, for example the “message” field of the Datadog payload can only take string values, JSON strings, and JSON/maps.
- incoming_data_types: custom, log, cluster_pattern_and_sample
- outgoing_data_types: datadog_payload
Most of the fields in the schema can be customized, except “event”
and “index”
. If not customized, it will use the default value:
Message: item["body"]
Tags: item["attributes"]
Source: item["resource"]["src_type"]
Source Name: item["resource"]["__src_name"]
Host: item["resource"]["host.name"]
Service:item["resource"]["__logical_source"]
Level: INFO
DataType: Log
If the specified field is missing from the incoming item, an empty string will be used as default value.
A timestamp is not defined because the default Datadog timestamp is used. You can select a field from the payload to use in Datadog using a remapper.
Example Configuration
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body"]
dd_service: "edgedelta"
dd_tags: item["attributes"]
Example Input log
{
"timestamp": "1581452773000000789",
"body": "Failed to do something",
"resource": {
"host.name": "host-1",
"ed.tag": "test-config",
"ed.org.id": "0481a213-....",
"__src_name": "source_name",
"src_type": "K8s",
"__logical_source": "logical_source",
"__short_src_name": "short_source_name",
"__group_name": "group_name",
"k8s.namespace.name": "edgedelta",
"k8s.pod.name": "api-deployment-d79fab72249c",
"k8s.container.name": "echo:latest",
"k8s.labels.app": "my-api",
},
"type": "log",
"attributes": {
"pod_id":"api-deployment-d79fab72249c-vtq9x",
"instance_id":"i-1234567890abcdef0",
"instance_name":"test-name",
"parsed": {
"timestamp": "2023/07/11 09:40:21",
"msg": "Failed to do something",
},
"original_timestamp": "2023/07/11 09:40:21",
}
}
Example Output
{
"message": "Failed to do something",
"host": "host-1",
"sourcename": "source_name",
"ddsource": "K8s",
"service": "edgedelta",
"level": "INFO",
"datatype": "Log",
"ddtags": {
"pod_id":"api-deployment-d79fab72249c-vtq9x",
"instance_id":"i-1234567890abcdef0",
"instance_name":"test-name",
"parsed": {
"timestamp": "2023/07/11 09:40:21",
"msg": "Failed to do something",
},
"original_timestamp": "2023/07/11 09:40:21",
},
}
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: datadog_mapper
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
dd_message
The dd_message
parameter is used to define which input item to use for the message field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
See Also:
dd_service
The dd_service
parameter is used to define which input item to use for the service
field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_service: item["resource"]["__logical_source"]
See Also:
dd_source
The dd_source
parameter is used to define which input item to use for the ddsource
field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_source: item["resource"]["__src_name"]
See Also:
dd_tags
The dd_tags
parameter is used to define which input item to use for the “ddtags” field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is required.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_tags: item["attributes"]
See Also:
Optional Parameters
dd_datatype
The dd_datatype
parameter is used to define the datatype
field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_datatype: Log
See Also:
dd_host
The dd_host
parameter is used to define which input item to use for the host
field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional but host
or hostname
must be set. Usually hostname
is not used if host
is specified, however, using both can cater for more advanced Datadog configurations.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_host: item["resource"]["host.name"]
See Also:
dd_hostname
The dd_hostname
parameter is used to define which input item to use for the hostname
field in the output Datadog payload instead of using the default agent tag. It is specified as a string and you can use CEL macros. It is optional but host
or hostname
must be set. Usually hostname
is not used if host
is specified, however, using both can cater for more advanced Datadog configurations.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_hostname: json(item["_parsed"]).hostname
See Also:
dd_level
The dd_level
parameter is used to define which input item to use for the level
field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_level: INFO
See Also:
dd_source_name
The dd_source_name
parameter is used to define which input item to use for the sourcename
field in the output Datadog payload. It is specified with CEL and you can use CEL macros. It is optional.
nodes:
- name: datadog_payload_mapping_example
type: datadog_mapper
dd_message: item["body""]
dd_source_name: item["resource"]["__src_name"]
See Also: