Send Data to Google Cloud Logging

Overview

You can send logs from an Edge Delta pipeline to Google’s Cloud Logging by adding a Google Cloud Logging destination node. See Google Cloud Logging with OTTL for configuration details.

Note: This guide applies to the google_cloud_logging_with_ottl_output node, which replaces the legacy google_cloud_logging_output node. The new node supports dynamic field construction using OTTL expressions.

Configure Cloud Logging

1. Service Account with JSON Credentials (Default Method):

   • Create a Service Account. Ensure that it has an IAM role that allows it to push to the Google Cloud API (Logging scope).
   • Add a new key for the newly created service account.
   • Download the credentials file containing the private key. Store it in a secure location. This is the path you specify with the credentials_path parameter when configuring the Edge Delta agent.

2. Service Account with Workload Identity (Recommended for GKE):

   • If using GKE, configure Edge Delta to utilize Workload Identity for authentication:
     1. Follow the steps provided by Google in the Workload Identity documentation.
     2. Ensure that you create an IAM allow policy that references the Kubernetes ServiceAccount. Specifically, grant the role roles/logging.logWriter to the IAM allow policy you just created. Below is an example on how to do it:

        gcloud projects add-iam-policy-binding projects/PROJECT_ID \
        --role=roles/logging.logWriter \
        --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME \
        --condition=None
        

Configure Kubernetes

If the Edge Delta agents are installed in a Kubernetes environment, depending on the authentication method chosen:

1. With JSON Credentials:

   • Create a secret using the credentials file you downloaded from Cloud Logging:

     kubectl create secret generic secret-name --from-file /path/to/credentials-file.json
     

     Note: Replace /path/to/credentials-file.json with the location and name of the downloaded credentials file.

2. Using Workload Identity:

   • Ensure that the necessary Kubernetes and IAM setup as detailed in the Workload Identity documentation is complete.
   • Omit the credentials_path parameter in the configuration.

Edge Delta Configuration

Next you configure the Google Cloud Logging destination node.

With JSON Credentials

nodes:
- name: google_cloud_logging_output
  type: google_cloud_logging_with_ottl_output
  log_name: attributes["log_name"]
  credentials_path: /etc/credentials/test-gcp-cl.json

Using Workload Identity

nodes:
- name: google_cloud_logging_output
  type: google_cloud_logging_with_ottl_output
  log_name: attributes["log_name"]
  # credentials_path: Not specified for Workload Identity

With Dynamic Resource Values

The resource_type_expression parameter is used to define an expression to fetch the resource type per incoming item dynamically. If not defined, "logging_log" will be used for all entries. In this example, the expression is configured as follows:

1. Check for resource_type Attribute: The expression begins by checking if the item has an attributes field and if that field contains a resource_type key using the has(item.attributes.resource_type) function. This ensures that the expression only attempts to access resource_type if it actually exists within item.attributes.
2. Check for Non-Empty resource_type: The expression further checks if item.attributes.resource_type is not an empty string item.attributes.resource_type != "". This ensures that the resource_type is not only present but also contains a meaningful value.
3. Conditional Assignment: If both conditions are met the expression evaluates to item.attributes.resource_type, using the value from the item’s attributes. If either condition fails, the expression defaults to "default_resource_type".

  - name: my_google_cloud_logging_output
    type: google_cloud_logging_with_ottl_output
    log_name: projects/my-project/logs/my-log-bucket
    credentials_path: /path/to/credentials.json
    resource_type_expression: 'has(item.attributes.resource_type) && item.attributes.resource_type != "" ? item.attributes.resource_type : "default_resource_type"'