Edge Delta Google Cloud Logging (OTTL) Destination

Send logs to Google Cloud Logging using OTTL expressions for dynamic fields.

3 minute read

Overview

You can send logs from an Edge Delta pipeline to Google Cloud Logging by adding a google_cloud_logging_with_ottl_output destination node. This node supports OTTL expressions to dynamically construct values for log_name, labels, and resource metadata based on incoming data.

incoming_data_types: log, cluster_pattern_and_sample, custom

Example Configuration

This configuration defines a Google Cloud Logging destination node named my_google_cloud_logging_output using the google_cloud_logging_with_ottl_output type. It dynamically sets the log_name and resource_type fields using OTTL expressions based on incoming event attributes. The node uses a JSON credentials file located at /etc/credentials/test-gcp-cl.json for authentication. It also appends Kubernetes-specific labels (k8s_namespace and k8s_pod_name) extracted from the event’s resource fields, and assigns structured resource metadata using the resource_labels_expression parameter.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]
    credentials_path: /etc/credentials/test-gcp-cl.json
    resource_type_expression: attributes["resource_type"]
    labels:
      - name: k8s_namespace
        path: resource["k8s.namespace.name"]
      - name: k8s_pod_name
        path: resource["k8s.pod.name"]
    resource_labels_expression: attributes["resource_labels"]

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: google_cloud_logging_with_ottl_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

log_name

Defines the logName for the Cloud Logging payload. This can be a static string or an OTTL expression.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]

Optional Parameters

credentials_path

Path to a Google Cloud service account JSON file.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]
    credentials_path: /path/to/credentials.json

resource_type_expression

An OTTL expression to define the resource_type per incoming item. Defaults to logging_log if not set.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]
    credentials_path: /path/to/credentials.json
    resource_type_expression: attributes["resource_type"]

labels

Adds labels to each log entry. Each label is defined by a name and an OTTL path expression.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]
    credentials_path: /path/to/credentials.json
    labels:
      - name: k8s_namespace
        path: resource["k8s.namespace.name"]
      - name: k8s_container_name
        path: resource["k8s.container.name"]
      - name: k8s_pod_name
        path: resource["k8s.pod.name"]

resource_labels_expression

Expression that returns a map of resource labels to attach to the log entry.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]
    credentials_path: /path/to/credentials.json
    resource_labels_expression: attributes["resource_labels"]

parallel_worker_count

Sets the number of workers writing in parallel. Default is 5.

nodes:
  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: attributes["log_name"]
    credentials_path: /path/to/credentials.json
    parallel_worker_count: 6

Authentication

You can authenticate using:

JSON Credentials File

credentials_path: /etc/credentials/my-gcp-creds.json

Make sure the associated service account has roles/logging.logWriter.

GKE Workload Identity

Omit credentials_path. Make sure the Kubernetes service account is bound correctly to a GCP IAM identity with log writing permissions.

Troubleshooting

Invalid or missing credentials – Check file path and content.
Permission errors – Ensure roles/logging.logWriter is granted.
Misconfigured log_name – Validate format or OTTL expression output.
Missing labels – Must be explicitly defined to be included.
Network/API access issues – Confirm API is enabled and reachable.
Still stuck? – Contact Edge Delta Support