Datadog K8s Pack

This is a Datadog K8s pack that ensures compatibility of log data with a Datadog destination.

  3 minute read  

Edge Delta Pipeline Pack for Datadog

Overview

The Datadog pack ensures compatibility of log data with a Datadog destination.

Pack Description

1. Data Ingestion

The data flow starts with the compound_input node. This node serves as the entry point into the pipeline, where it begins processing the incoming log data.

- name: compound_input
  type: compound_input

2. Tag Enrichment

Next, logs flow into the upsert node, which is an Output Transform node. This node enhances the logs by adding or updating various tag fields in the attributes section.

- name: upsert
  type: output_transform
  transformations:
  - field_path: item["attributes"]["tags"]["cluster_name"]
    operation: upsert
    value: item["resource"]["ed.tag"]
  - field_path: item["attributes"]["tags"]["container_id"]
    operation: upsert
    value: item["resource"]["container.id"]
  - field_path: item["attributes"]["tags"]["container_name"]
    operation: upsert
    value: item["resource"]["k8s.container.name"]
  - field_path: item["attributes"]["tags"]["dirname"]
    operation: upsert
    value: regex_capture(item["resource"]["ed.filepath"], "(?P<dirname>.+)/[0-9a-zA-Z].log").dirname
  - field_path: item["attributes"]["tags"]["filename"]
    operation: upsert
    value: regex_capture(item["resource"]["ed.filepath"], ".+/(?P<filename>[0-9a-zA-Z]+.log)").filename
  - field_path: item["attributes"]["tags"]["image_name"]
    operation: upsert
    value: regex_capture(item["resource"]["container.image.name"], "(?P<image_name>.+):").image_name
  - field_path: item["attributes"]["tags"]["image_tag"]
    operation: upsert
    value: regex_capture(item["resource"]["container.image.name"], ":(?P<image_tag>.+)").image_tag
  - field_path: item["attributes"]["tags"]["kube_cluster_name"]
    operation: upsert
    value: item["resource"]["ed.tag"]
  - field_path: item["attributes"]["tags"]["kube_container_name"]
    operation: upsert
    value: item["resource"]["k8s.container.name"]
  - field_path: item["attributes"]["tags"]["kube_deployment"]
    operation: upsert
    value: item["resource"]["k8s.deployment.name"]
  - field_path: item["attributes"]["tags"]["kube_namespace"]
    operation: upsert
    value: item["resource"]["k8s.namespace.name"]
  - field_path: item["attributes"]["tags"]["kube_node"]
    operation: upsert
    value: item["resource"]["k8s.node.name"]
  - field_path: item["attributes"]["tags"]["kube_service"]
    operation: upsert
    value: item["resource"]["k8s.deployment.name"]
  - field_path: item["attributes"]["tags"]["pod_name"]
    operation: upsert
    value: item["resource"]["k8s.pod.name"]
  - field_path: item["attributes"]["tags"]["service"]
    operation: upsert
    value: item["resource"]["k8s.deployment.name"]
  - field_path: item["attributes"]["tags"]["short_image"]
    operation: upsert
    value: regex_capture(item["attributes"]["image_name"], "/(?P<short_image>[a-zA-Z0-9]+)$").short_image
  - field_path: item["attributes"]["tags"]["source"]
    operation: upsert
    value: item["resource"]["k8s.deployment.name"]
  - field_path: item["attributes"]["tags"]["display_container_name"]
    operation: upsert
    value: merge(item["attributes"]["container_name"], item["attributes"]["pod_name"])

It uses multiple transformations to extract values from the resource fields and insert them into attributes tags:

  • cluster_name tag: The value is taken from the resource field ed.tag, containing the fleet name.
  • container_id" tag: The value is extracted from container.id, tagging the log entry with the container ID.
  • container_name tag: Extracts the Kubernetes container name from the resource and assigns it as a tag.
  • dirname tag: Uses regex_capture to extract the directory name from the ed.filepath.
  • filename tag: Uses regex_capture to isolate the filename from the filepath, capturing filenames ending with .log.
  • image_name tag: The regex captures the image name from a versioned image identifier, extracting all content before the colon.
  • image_tag tag: Captures the version tag from the image name, extracting content after the colon.
  • kube_cluster_name tag: Similar to cluster name upsert directly using ed.tag.
  • kube_container_name tag: Tags the log with the container’s Kubernetes name.
  • kube_deployment tag: Extracts and assigns the deployment name from resource details.
  • kube_namespace tag: Extracts the namespace from the resource and assigns it as a tag.
  • kube_node tag: Tags the log with the node name from Kubernetes resource data.
  • kube_service tag: Uses the deployment name as the service tag.
  • pod_name tag: Extracts and assigns the name of the pod from Kubernetes resource attributes.
  • service tag: Uses the deployment name attribute as the service tag.
  • short_image tag: Uses regex_capture to get the short image name by capturing the last segment in a path-like structure.
  • source tag: Assigns the deployment name as the source, useful for application identification.
  • display_container_name tag: Uses the merge function to create a composite display name from the container and pod names, aiding in context.

These transformations help in enriching logs with contextual tags, making it easier to search and filter the log data later.

3. Datadog Integration

The enriched logs then pass to the datadog_mapper node, which is a Datadog Mapper node. This node structures the log data into a format compatible with Datadog, assigning message, tags, source, host, service, and level to Datadog fields.

  - name: datadog_mapper
    type: datadog_mapper
    dd_message: item["body"]
    dd_tags: item["attributes"]["tags"]
    dd_source: item["attributes"]["tags"]["source"]
    dd_host: item["attributes"]["tags"]["kube_node"]
    dd_service: item["attributes"]["tags"]["service"]
    dd_level: item["severity_text"]
    dd_datatype: '"log"'

This node allows seamless integration with Datadog for log ingestion, enabling real-time monitoring, visualization, and alerting based on log data.

4. Data Output

Finally, the processed logs reach the compound_output node, which acts as the terminal point for the pipeline, where logs are outputted to your desired destination.

- name: compound_output
  type: compound_output

Sample Input

Sample K8s Log