Edge Delta Google Cloud Logging Destination

Archive using a Google Cloud Logging destination.

4 minute read

Overview

You can send logs from an Edge Delta fleet to Google’s Cloud Logging by adding a Google Cloud Logging destination node to your Fleet’s pipeline.

incoming_data_types: log, cluster_pattern_and_sample, custom

Configure Cloud Logging

See Send Data to Google Cloud Logging.

Edge Delta Configuration

Next you configure the Google Cloud Logging destination node.

With JSON Credentials

nodes:
- name: google_cloud_logging_output
  type: google_cloud_logging_output
  log_name: projects/edgedelta/logs/my_test
  credentials_path: /etc/credentials/test-gcp-cl.json

Using Workload Identity

nodes:
- name: google_cloud_logging_output
  type: google_cloud_logging_output
  log_name: projects/edgedelta/logs/my_test
  # credentials_path: Not specified for Workload Identity

With Dynamic Resource Values

The resource_type_expression parameter is used to define an expression to fetch the resource type per incoming item dynamically. If not defined, "logging_log" will be used for all entries. In this example, the expression is configured as follows:

Check for resource_type Attribute: The expression begins by checking if the item has an attributes field and if that field contains a resource_type key using the has(item.attributes.resource_type) function. This ensures that the expression only attempts to access resource_type if it actually exists within item.attributes.
Check for Non-Empty resource_type: The expression further checks if item.attributes.resource_type is not an empty string item.attributes.resource_type != "". This ensures that the resource_type is not only present but also contains a meaningful value.
Conditional Assignment: If both conditions are met the expression evaluates to item.attributes.resource_type, using the value from the item’s attributes. If either condition fails, the expression defaults to "default_resource_type".

  - name: my_google_cloud_logging_output
    type: google_cloud_logging_output
    log_name: projects/my-project/logs/my-log-bucket
    credentials_path: /path/to/credentials.json
    resource_type_expression: 'has(item.attributes.resource_type) && item.attributes.resource_type != "" ? item.attributes.resource_type : "default_resource_type"'

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: google_cloud_logging_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

log_name

The log_name parameter is used to define the logName field of the log entry payload. The log_name must follow one of the following formats:

projects/[PROJECT_ID]/logs/[LOG_ID]
organizations/[ORGANIZATION_ID]/logs/[LOG_ID]
billingAccounts/[BILLING_ACCOUNT_ID]/logs/[LOG_ID]
folders/[FOLDER_ID]/logs/[LOG_ID]

A log_name is required.

nodes:
- name: my_google_cloud_logging_output
  type: google_cloud_logging_output
  log_name: projects/my-project/logs/my-log-bucket
  credentials_path: /path/to/credentials.json

Optional Parameters

credentials_path

The credentials_path parameter is used to define the location of your Google service account credentials. It is an optional field.

nodes:
- name: my_google_cloud_logging_output
  type: google_cloud_logging_output
  log_name: projects/my-project/logs/my-log-bucket
  credentials_path: /path/to/credentials.json

labels

The labels parameter is used to define labels to append to logs. This is useful for differentiating between log sources. Labels are defined with a name for the field, and a path to the source field containing the label value. See here for information on how to reference fields. You can also use CEL macros so the value can be extracted from data. Labels are optional.

nodes:
- name: my_google_cloud_logging_output
  type: google_cloud_logging_output
  log_name: projects/my-project/logs/my-log-bucket
  credentials_path: /path/to/credentials.json
  labels:
    - name: k8s_namespace
      path: item["resource"]["k8s.namespace.name"]
    - name: k8s_container_name
      path: item["resource"]["k8s.container.name"]
    - name: k8s_pod_name
      path: item["resource"]["k8s.pod.name"]
    - name: k8s_deployment_name
      path: item["resource"]["k8s.deployment.name"]
    - name: k8s_daemonset_name
      path: item["resource"]["k8s.daemonset.name"]
    - name: k8s_statefulset_name
      path: item["resource"]["k8s.statefulset.name"]
    - name: k8s_replicaset_name
      path: item["resource"]["k8s.replicaset.name"]
    - name: k8s_cronjob_name
      path: item["resource"]["k8s.cronjob.name"]
    - name: k8s_job_name
      path: item["resource"]["k8s.job.name"]

Note: The following Edge Delta fields are not passed by the output unless explicitly set as labels:

item["resource"]["__group_name"]
item["resource"]["__logical_source"]
item["resource"]["__short_src_name"]
item["resource"]["__src_name"]

resource_type_expression

The resource_type_expression parameter is used to specify an expression to dynamically determine the resource_type value. It is specified as a string and is optional.

- name: my_google_cloud_logging_output
  type: google_cloud_logging_output
  log_name: projects/my-project/logs/my-log-bucket
  credentials_path: /path/to/credentials.json
  resource_type_expression: 'has(item.attributes.resource_type) && item.attributes.resource_type != "" ? item.attributes.resource_type : "default_resource_type"'

See Also:

Knowledge Library

Troubleshooting

Check OAuth2 Credentials: Verify that the OAuth2 credentials being used are correct and have not expired. Ensure that the credentials are properly configured in your Edge Delta setup.
Permissions and Roles: Ensure that the service account associated with the OAuth2 credentials has the necessary permissions to write logs to Google Cloud Logging. The service account should have roles like roles/logging.logWriter. Click IAM & Admin - IAM to check and update the permissions in the Google Cloud Console.
Token Scopes: Ensure that the OAuth2 token request includes the correct scopes for Google Cloud Logging. The required scope is typically https://www.googleapis.com/auth/logging.write.
API Access: Click APIs & Services - Library to verify that the Google Cloud Logging API is enabled for your project in the Google Cloud Console. Check if there are any restrictions or firewall rules that might be blocking the API requests from Edge Delta.
Service Account Key: If you are using a JSON key file for the service account, ensure that it is correctly referenced and accessible by Edge Delta.
Network Configuration: Check for any network issues or restrictions that might be preventing Edge Delta from reaching the Google Cloud Logging endpoint.
Contact Edge Delta Support.