Edge Delta Google Cloud Logging Output
4 minute read
Overview
You can send logs from an Edge Delta fleet to Google’s Cloud Logging by adding a Google Cloud Logging output node to your Fleet’s pipeline.
- incoming_data_types: log, cluster_pattern_and_sample, custom
Configure Cloud Logging
-
Service Account with JSON Credentials (Default Method):
- Create a Service Account. Ensure that it has an IAM role that allows it to push to the Google Cloud API (Logging scope).
- Add a new key for the newly created service account.
- Download the credentials file containing the private key. Store it in a secure location. This is the path you specify with the
credentials_path
parameter when configuring the Edge Delta agent.
-
Service Account with Workload Identity (Recommended for GKE):
- If using GKE, configure Edge Delta to utilize Workload Identity for authentication:
- Follow the steps provided by Google in the Workload Identity documentation.
- Ensure that you create an IAM allow policy that references the Kubernetes ServiceAccount. Specifically, grant the role
roles/logging.logWriter
to the IAM allow policy you just created. Below is an example on how to do it:gcloud projects add-iam-policy-binding projects/PROJECT_ID \ --role=roles/logging.logWriter \ --member=principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/NAMESPACE/sa/KSA_NAME \ --condition=None
- If using GKE, configure Edge Delta to utilize Workload Identity for authentication:
Configure Kubernetes
If the Edge Delta agents are installed in a Kubernetes environment, depending on the authentication method chosen:
-
With JSON Credentials:
- Create a secret using the credentials file you downloaded from Cloud Logging:
kubectl create secret generic secret-name --from-file /path/to/credentials-file.json
Note: Replace
/path/to/credentials-file.json
with the location and name of the downloaded credentials file.
- Create a secret using the credentials file you downloaded from Cloud Logging:
-
Using Workload Identity:
- Ensure that the necessary Kubernetes and IAM setup as detailed in the Workload Identity documentation is complete.
- Omit the
credentials_path
parameter in the configuration.
Example Edge Delta Configuration
With JSON Credentials
nodes:
- name: google_cloud_logging_output
type: google_cloud_logging_output
log_name: projects/edgedelta/logs/my_test
credentials_path: /etc/credentials/test-gcp-cl.json
Using Workload Identity
nodes:
- name: google_cloud_logging_output
type: google_cloud_logging_output
log_name: projects/edgedelta/logs/my_test
# credentials_path: Not specified for Workload Identity
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: google_cloud_logging_output
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
log_name
The log_name
parameter is used to define the logName
field of the log entry payload. The log_name
must follow one of the following formats:
projects/[PROJECT_ID]/logs/[LOG_ID]
organizations/[ORGANIZATION_ID]/logs/[LOG_ID]
billingAccounts/[BILLING_ACCOUNT_ID]/logs/[LOG_ID]
folders/[FOLDER_ID]/logs/[LOG_ID]
A log_name is required.
nodes:
- name: my_google_cloud_logging_output
type: google_cloud_logging_output
log_name: projects/my-project/logs/my-log-bucket
credentials_path: /path/to/credentials.json
Optional Parameters
credentials_path
The credentials_path
parameter is used to define the location of your Google service account credentials. It is an optional field.
nodes:
- name: my_google_cloud_logging_output
type: google_cloud_logging_output
log_name: projects/my-project/logs/my-log-bucket
credentials_path: /path/to/credentials.json
labels
The labels
parameter is used to define labels to append to logs. This is useful for differentiating between log sources. Labels are defined with a name for the field, and a path to the source field containing the label value. See here for information on how to reference fields. You can also use CEL macros so value can be extracted from data. Labels are optional.
nodes:
- name: my_google_cloud_logging_output
type: google_cloud_logging_output
log_name: projects/my-project/logs/my-log-bucket
credentials_path: /path/to/credentials.json
labels:
- name: k8s_namespace
path: item["resource"]["k8s.namespace.name"]
- name: k8s_container_name
path: item["resource"]["k8s.container.name"]
- name: k8s_pod_name
path: item["resource"]["k8s.pod.name"]
- name: k8s_deployment_name
path: item["resource"]["k8s.deployment.name"]
- name: k8s_daemonset_name
path: item["resource"]["k8s.daemonset.name"]
- name: k8s_statefulset_name
path: item["resource"]["k8s.statefulset.name"]
- name: k8s_replicaset_name
path: item["resource"]["k8s.replicaset.name"]
- name: k8s_cronjob_name
path: item["resource"]["k8s.cronjob.name"]
- name: k8s_job_name
path: item["resource"]["k8s.job.name"]
See Also:
Troubleshooting
- Check OAuth2 Credentials: Verify that the OAuth2 credentials being used are correct and have not expired. Ensure that the credentials are properly configured in your Edge Delta setup.
- Permissions and Roles: Ensure that the service account associated with the OAuth2 credentials has the necessary permissions to write logs to Google Cloud Logging. The service account should have roles like
roles/logging.logWriter
. Click IAM & Admin - IAM to check and update the permissions in the Google Cloud Console. - Token Scopes: Ensure that the OAuth2 token request includes the correct scopes for Google Cloud Logging. The required scope is typically
https://www.googleapis.com/auth/logging.write
. - API Access: Click APIs & Services - Library to verify that the Google Cloud Logging API is enabled for your project in the Google Cloud Console. Check if there are any restrictions or firewall rules that might be blocking the API requests from Edge Delta.
- Service Account Key: If you are using a JSON key file for the service account, ensure that it is correctly referenced and accessible by Edge Delta.
- Network Configuration: Check for any network issues or restrictions that might be preventing Edge Delta from reaching the Google Cloud Logging endpoint.
- Contact Edge Delta Support.