Edge Delta Azure Event Hub Source

Overview

The Azure Event Hub Source node reads data from an Azure Event Hub. It can operate on specific partitions or use the Event Processor Host model to distribute partitions across multiple agents.

• outgoing_data_types: log

Example Configuration

This configuration connects Edge Delta to an Azure Event Hub and listens for streaming events using the default consumer group.

nodes:
  - name: my_azure_eventhub
    type: eventhub_input
    connection_string: "Endpoint=sb://..."
    consumer_group: "$Default"

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: eventhub_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

connection_string

The connection_string parameter specifies how to connect to Azure Event Hub. Obtain it from the Azure portal.

nodes:
  - name: my_azure_eventhub
    type: eventhub_input
    connection_string: "Endpoint=sb://..."

Optional Parameters

consumer_group

Specify the Event Hub consumer group. Defaults to $Default.

nodes:
  - name: my_azure_eventhub
    type: eventhub_input
    connection_string: "Endpoint=sb://..."
    consumer_group: "$Default"

partition_ids

List the partition IDs to consume. When omitted, the Event Processor Host model balances partitions across agents.

nodes:
  - name: my_azure_eventhub
    type: eventhub_input
    connection_string: "Endpoint=sb://..."
    partition_ids:
      - 0
      - 1

checkpoint_directory

Directory path to persist sequence numbers when using specific partitions.

nodes:
  - name: my_azure_eventhub
    type: eventhub_input
    connection_string: "Endpoint=sb://..."
    partition_ids:
      - 0
    checkpoint_directory: "/var/eventhub-checkpoint/"

storage_account_name, storage_account_key, storage_container_name

Provide a storage account and container for lease and checkpoint management when partition_ids are not set.

nodes:
  - name: my_azure_eventhub
    type: eventhub_input
    connection_string: "Endpoint=sb://..."
    storage_account_name: mystorageacc
    storage_account_key: "*****"
    storage_container_name: "edgedelta-eventhub-container"

source_metadata

The source_metadata parameter is used to define which detected resources and attributes to add to each data item as it is ingested by the Edge Delta agent. In the GUI you can select:

• Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
• Default: This option includes the required resources and attributes plus those selected by Edge Delta
• High: This option includes the required resources and attributes along with a larger selection of common optional fields.
• Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.