Troubleshooting Datadog Tags

Overview

This guide helps you diagnose and fix issues with tags when sending data from Edge Delta to Datadog. Tags in Datadog use a comma-separated key:value format (for example, env:prod,service:api). How Edge Delta constructs those tags depends on how you configure your pipeline.

For destination setup, see Datadog Destination. For preparation steps, see Send Data to Datadog.

How tags reach Datadog

The Datadog destination node automatically constructs ddtags from the data item. Tags come from several sources, combined additively:

• Edge Delta internal tags: ed_integration:true, edgedelta_datatype:log, sourceType:k8s, sourceName:my-source, tag:my-pipeline
• Source attributes: Kubernetes metadata (namespace, pod, container, deployment), ECS metadata (cluster, task family), or Docker metadata (container name, image)
• Item attributes: All key-value pairs in the item’s attributes field
• Custom tag templates: Any custom_tags configured on the destination node
• Static tags: Tags set via the ED_CUSTOM_TAGS environment variable

To control which tags appear in Datadog, shape item attributes before the destination node using OTTL statements in a processor. The destination node then converts those attributes into the ddtags string.

Static tags with ED_CUSTOM_TAGS

The ED_CUSTOM_TAGS environment variable attaches static attributes to all items processed by the agent. These attributes flow through to the Datadog destination as tags.

env:
  - name: ED_CUSTOM_TAGS
    value: "cluster:prod_us_west_2|provider:aws|region:us_west_2"

Format: key1:value1|key2:value2|key3:value3

Tags missing in Datadog

Attributes not set before the destination

Symptom: Logs appear in Datadog but expected tags like env, service, or version are missing from ddtags.

Cause: The Datadog destination converts item attributes to tags. If the attributes you expect are not present on the item when it reaches the destination, they will not appear as tags.

Diagnosis: Use Live Tail to inspect the data item at the point just before the Datadog destination. Check whether the expected fields exist in attributes or resource.

Solution: Use an OTTL transform processor to set the attributes you need:

- name: Enrich for Datadog
  type: sequence
  processors:
  - type: ottl_transform
    statements: |-
      set(attributes["env"], resource["deployment.environment"])
      set(attributes["service"], resource["service.name"])
      set(attributes["version"], resource["service.version"])      

ED_CUSTOM_TAGS not taking effect

Symptom: Static tags configured via ED_CUSTOM_TAGS do not appear on logs in Datadog.

Cause: The environment variable may not be set correctly, or the format is wrong.

Diagnosis: Verify the variable is set in the agent’s environment:

# Kubernetes
kubectl get daemonset -n edgedelta edgedelta -o jsonpath='{.spec.template.spec.containers[0].env}' | jq .

Solution: Ensure the format uses pipe-separated key:value pairs with no spaces around the pipes:

env:
  - name: ED_CUSTOM_TAGS
    value: "env:prod|region:us-west-2|team:platform"

Lambda tags not appearing

Symptom: Logs from AWS Lambda arrive in Datadog without the Lambda function’s resource tags (such as env, service, or version).

Cause: Lambda tag forwarding is opt-in. Edge Delta does not fetch AWS resource tags unless you explicitly enable it.

Solution: Set the appropriate environment variables on your Lambda forwarder or extension:

Note: ED_FORWARD_SOURCE_TAGS builds the source ARN from the log group name. For Lambda sources, the log group must follow the /aws/lambda/<lambda_name> naming convention.

For more details, see Lambda Forwarder and Lambda Extension.

Missing IAM permissions

Symptom: Lambda tag environment variables are enabled but tags still do not appear.

Cause: The Lambda execution role lacks the IAM permissions required to fetch resource tags.

Diagnosis: Check CloudWatch Logs for the forwarder Lambda. Permission errors appear as AccessDeniedException entries.

Solution: Add the required permissions to the Lambda execution role:

{
  "Effect": "Allow",
  "Action": [
    "tag:GetResources",
    "lambda:GetFunctionConfiguration",
    "lambda:ListTags"
  ],
  "Resource": "*"
}

Unexpected or extra tags in Datadog

Edge Delta internal tags appearing

Symptom: Logs in Datadog contain tags you did not configure, such as ed_integration:true, edgedelta_datatype:log, sourceType:k8s, or sourceName:my-deployment.

Cause: The Datadog destination automatically adds Edge Delta metadata as tags. This is by design and helps identify Edge Delta data in Datadog.

Solution: These internal tags are part of the default behavior and cannot be disabled on the Datadog destination node.

All attributes promoted to tags

Symptom: Every attribute on the data item appears as a tag in Datadog, including fields you do not want as tags (such as parsed message fields or timestamps).

Cause: The Datadog destination converts all item attributes to tags. If your pipeline enriches items with many attributes, all of them become Datadog tags.

Solution: Use an OTTL transform processor to remove unwanted attributes before the destination:

- name: Clean Attributes
  type: sequence
  processors:
  - type: ottl_transform
    statements: |-
      delete_key(attributes, "parsed_timestamp")
      delete_key(attributes, "raw_message")      

Alternatively, restructure your pipeline so that only the attributes you want as tags are present when items reach the Datadog destination.

Tags have wrong format

Nested maps produce dot-separated keys

Symptom: Tags in Datadog have keys like parsed.timestamp, parsed.msg, or nested.field.name instead of simple keys.

Cause: When item attributes contain nested maps, Edge Delta flattens the structure using dots as separators before converting to key:value format.

For example, this attributes structure:

{
  "env": "prod",
  "parsed": {
    "timestamp": "2025-07-11T09:40:21Z",
    "msg": "Request failed"
  }
}

Produces tags: env:prod,parsed.msg:Request failed,parsed.timestamp:2025-07-11T09:40:21Z

Solution: Use an OTTL transform processor to flatten or remove nested attributes before the destination:

- name: Flatten for Datadog
  type: sequence
  processors:
  - type: ottl_transform
    statements: |-
      set(attributes["log_msg"], attributes["parsed"]["msg"])
      delete_key(attributes, "parsed")      

Boolean and numeric values converted to strings

Symptom: Tag values appear as true, false, or raw numbers instead of expected string values.

Cause: Edge Delta converts all non-string values to their string representation when building the tag string. This is expected behavior and matches Datadog’s tag format (all tag values are strings).

Datadog-to-Datadog passthrough

Tags duplicated or prefixed incorrectly

Symptom: When receiving data from a Datadog Agent input and sending to a Datadog destination, tags appear with a datadog.tags. prefix (for example, datadog.tags.env instead of env) or are duplicated.

Cause: In versions before v2.10.0, the Datadog output incorrectly added all attributes as tags when using a Datadog input, which caused duplication. The datadog.tags. prefix is an internal representation that should be stripped at the output layer.

Solution: Upgrade to agent v2.10.0 or later. In current versions, Edge Delta automatically strips the datadog.tags. prefix when sending to a Datadog destination (for example, datadog.tags.env becomes env).

Quick reference

See also

• Datadog Destination for destination configuration
• Send Data to Datadog for preparation steps
• Custom Processor for attribute enrichment with OTTL statements
• Environment Variables for static tag configuration
• Lambda Forwarder for AWS Lambda tag forwarding
• Lambda Extension for Lambda extension tag enrichment