Use Edge Delta to Ingest from a GCS Source

Configure Google Cloud Storage object-change notifications, a Pub/Sub subscription, and a service account so the Edge Delta agent can ingest objects from a GCS bucket.

  2 minute read  

Overview

The GCS source node allows the Edge Delta agent to read log objects from a Google Cloud Storage bucket. The agent does not scan the bucket directly. Instead, GCS publishes an object-change notification to a Pub/Sub topic whenever an object is written, and the agent pulls those notifications from a Pub/Sub subscription and downloads each referenced object.

Set up the following in Google Cloud before you configure the node.

Configure GCS object notifications

Configure your bucket to publish object-change notifications to a Pub/Sub topic. Using the Google Cloud CLI:

gcloud storage buckets notifications create gs://YOUR_BUCKET \
  --topic=YOUR_TOPIC \
  --event-types=OBJECT_FINALIZE

OBJECT_FINALIZE fires when a new object is written. See Pub/Sub notifications for Cloud Storage for the full set of event types and options.

Create a Pub/Sub subscription

Create a subscription on the topic that receives the notifications. The agent pulls object-change messages from this subscription.

gcloud pubsub subscriptions create YOUR_SUBSCRIPTION --topic=YOUR_TOPIC

Use the subscription ID as the sub_id parameter and the project ID as the project_id parameter on the node.

Create a service account

Create a service account for the agent and grant it:

  • roles/pubsub.subscriber on the subscription, so it can pull object-change notifications.
  • roles/storage.objectViewer on the bucket, so it can download objects.

Create a JSON key for the service account. Provide the key to the node with either the key parameter (inline JSON) or the key_path parameter (a path to the key file on the agent host). See Create and manage service account keys for details.

Next, configure a pipeline with a GCS source node.

See Also