GCP Authentication for Edge Delta

Authenticate Edge Delta agents to Google Cloud services using service account keys or Workload Identity.

  3 minute read  

Overview

Edge Delta integrates with several Google Cloud services, including BigQuery, Cloud Storage, Cloud Logging, Google SecOps, and Pub/Sub. Each integration requires the agent to authenticate with the target service.

There are three ways to provide GCP authentication:

MethodEnvironmentKey benefit
Service account JSON keyAnyWorks everywhere
GKE Workload IdentityGKENo key files to manage
Compute Engine default credentialsGCE VMsAutomatic, no configuration

Required IAM Roles

Each destination requires specific IAM roles on the GCP service account. Grant only the roles your pipeline needs.

DestinationRequired IAM rolesSetup guide
BigQueryroles/bigquery.dataEditor, roles/bigquery.jobUserSend Data to BigQuery
Cloud StorageStorage Admin + Storage HMAC Key AdminSend Data to GCS
Cloud Loggingroles/logging.logWriterSend Data to Cloud Logging
Google SecOpsChronicle ingestion permissionsGoogle SecOps Destination

Option 1: Service Account JSON Key

Use a JSON key file when the agent runs outside GKE, or when Workload Identity is not available.

The general workflow is:

  1. Create a GCP service account in the project that owns the target resource.
  2. Grant the service account the IAM roles listed in the table above for your destination.
  3. Generate and download a JSON key file for the service account.
  4. Make the key file accessible to the Edge Delta agent:
    • Kubernetes — store the file as a Kubernetes secret and mount it into the agent pod.
    • Linux / VM — place the file on disk with restricted permissions (chmod 600).
  5. Set the credentials_path parameter in your destination node configuration to point to the key file.
nodes:
  - name: my_destination
    type: <destination_type>
    credentials_path: /etc/credentials/my-gcp-creds.json
    # ... other destination parameters

Each destination’s setup guide includes detailed steps for creating secrets and mounting credentials. See the links in the Required IAM Roles table.

Security: Rotate service account keys periodically and avoid storing them in version control. For Kubernetes deployments, use Kubernetes Secrets or an external secrets manager.

If the agent runs in Google Kubernetes Engine (GKE), Workload Identity lets pods authenticate as a GCP service account without key files. The Kubernetes service account is bound to a GCP IAM service account, and credentials are provided automatically by the GKE metadata server.

The general workflow is:

  1. Enable Workload Identity on your GKE cluster.
  2. Create a GCP service account with the required IAM roles.
  3. Bind the Edge Delta Kubernetes service account to the GCP service account using an IAM policy binding.
  4. Annotate the Kubernetes service account with the GCP service account email.
  5. Omit the credentials_path parameter in the destination node configuration.
nodes:
  - name: my_destination
    type: <destination_type>
    # credentials_path is omitted — Workload Identity provides credentials
    # ... other destination parameters

See Google’s Workload Identity documentation for the detailed GKE setup steps. The Send Data to BigQuery and Send Data to Cloud Logging pages include Edge Delta-specific examples.

Option 3: Compute Engine Default Credentials

When the agent runs on a GCE VM, it can use the VM’s attached service account automatically. No credentials file or special configuration is required, as long as the VM’s service account has the IAM roles needed by your destination.

Omit credentials_path in the destination node configuration. The agent uses Application Default Credentials (ADC), which automatically discover the VM’s identity.

See Also