Edge Delta Google Cloud Pub/Sub Source

Read data from Google Cloud Pub/Sub.

  3 minute read  

Beta

Overview

The Pub/Sub Source node allows the Edge Delta agent to read data from Google Cloud pub/sub topics. This node is essential for ingesting log data published on pub/sub and processing it within the Edge Delta ecosystem.

  • outgoing_data_types: log

Example Configuration 1

This configuration retrieves messages using a directly provided authentication key.

nodes:
- name: google_pubsub_with_key
  type: pubsub_input
  project_id: "projectID1"
  sub_id: "subID1"
  key: '{"pubsub_key": "key_123"}'

Example Configuration 2

This configuration pulls messages using a file path to an authentication key.

nodes:
- name: google_pubsub_with_key_path
  type: pubsub_input
  project_id: "projectID2"
  sub_id: "subID2"
  key_path: "pubsub_key_path"

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: pubsub_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

project_id

The project_id parameter specifies the Project ID of the topic/subscription in Google Cloud. It is a string and is required.

nodes:
- name: <node name>
  type: pubsub_input
  project_id: <your project ID>

sub_id

The sub_id parameter indicates the Pub/Sub subscription from which messages will be pulled. It is a string and is required

nodes:
- name: <node name>
  type: pubsub_input
  sub_id: <your subscription ID>

Optional Parameters

key

The key parameter is a password value required for authentication if key_path is not provided. This key will be used by agents to authenticate calls to Pub/Sub.

nodes:
- name: <node name>
  type: pubsub_input
  key: '{"pubsub_key": "your_key"}'

key_path

The key_path parameter specifies the file path for the key used in the authentication process. It is an alternative to the key parameter.

nodes:
- name: <node name>
  type: pubsub_input
  key_path: "/path/to/keyfile"

source_metadata

The source_metadata parameter is used to define which detected resources and attributes to add to each data item as it is ingested by the Edge Delta agent. In the GUI you can select:

  • Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
  • Default: This option includes the required resources and attributes plus those selected by Edge Delta
  • High: This option includes the required resources and attributes along with a larger selection of common optional fields.
  • Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.