Edge Delta with Azure Event Hubs

Overview

You can configure the Edge Delta agent to consume logs from Azure Event Hubs.

To connect Edge Delta to Azure Event Hubs you configure it with the connection_string provided by Azure, and you specify a consumer group. The connection string contains the fully qualified domain name of the Event Hubs namespace as well as the shared key name and shared key value.

You can specify specific partitions to consume from using the partition_ids parameter. If you don’t specify a partition, the Event Processor Host approach is used to load balance partitions between consumers. In this instance, the checkpoint_dir parameter is used to specify a location to persist last read sequence numbers.

You can alternatively use Azure storage as a lease and checkpoint backend by specifying the storage account parameters storage_account_name, storage_account_key, and storage_container_name. If you use a storage account you do not configure any partition IDs. When a storage account is used, multiple agents communicate via blob objects to load balance the Event Hubs partitions evenly and keep track of last retrieved event offset.

Example

The following example illustrates Event Hubs inputs. The first uses partition IDs, the second uses the Event Processor Host approach with a checkpoint directory, while the third input is configured with a storage account.

inputs:
  eventhubs:
    - labels: "errorcheck"
      connection_string: "<connection string>"
      consumer_group: "$Default"
      partition_ids: "0,1,2,3"
      checkpoint_dir: "/var/eventhub-checkpoint/"
    - labels: "errorcheck2"
      connection_string: "<connection string>"
      consumer_group: "$Default"
      checkpoint_dir: "/var/eventhub-checkpoint/"
    - labels: "errorcheck3"
      connection_string: "<connection string>"
      consumer_group: "$Default"
      storage_account_name: "mystorageacc"
      storage_account_key: "*****"
      storage_container_name: "<container name>"      

Required Parameters

labels

The labels parameter specifies one or more names for the input. You refer to one of these labels in other places, for example to refer to a specific input in a workflow. Labels must be unique within the inputs section. It is a yaml list element so it begins with a - and a space followed by the string. A label is required for an input.

inputs:
  eventhubs:
    - labels: "<input name>"

connection_string

The connection_string parameter specifies the fully qualified domain name of the Event Hubs namespace and the shared key name and value. It is provided by Azure. A connection_string is required for an Event Hubs input.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"

consumer_group

The consumer_group parameter is used to specify an existing Event Hubs consumer group you want the Edge Delta agent to belong to. It is specified with a string and is a required parameter for an Event Hubs input.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"
      consumer_group: "<consumer group name>"

Optional Parameters

partition_ids

The partition_ids parameter is used to define which Event Hubs partitions to consume. If no partition_ids are specified, the Event Processor Host approach is used for load balancing partitions between consumers. Partition IDs are specified using a string. A partition_ids parameter is optional.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"
      partition_ids: "<partition 1>, <partition 2>"

checkpoint_dir

The checkpoint_dir parameter is used to specify a directory for persisting last read sequence numbers. It is used when no partition_ids are specified. It is specified as a string and is optional.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"
      checkpoint_dir: "<directory to persist>"

storage_account_name

The storage_account_name parameter specifies the account name if Azure is used for lease and checkpoint storage. It is used with storage_account_key and storage_container_name. It is specified with a string and is optional.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"
      storage_account_name: "<account name>"
      storage_account_key: "<account key>"
      storage_container_name: "<container name>"

storage_account_key

The storage_account_key parameter specifies the account credentials if Azure is used for lease and checkpoint storage. It is used with storage_account_name and storage_container_name. It is specified with a string and is optional.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"
      storage_account_name: "<account name>"
      storage_account_key: "<account key>"
      storage_container_name: "<container name>" 

storage_container_name

The storage_container_name parameter specifies container if Azure is used for lease and checkpoint storage. It is used with storage_account_name and storage_account_key. It is specified with a string and is optional.

inputs:
  eventhubs:
    - labels: "<input label>"
      connection_string: "<Event Hubs connection string >"
      storage_account_name: "<account name>"
      storage_account_key: "<account key>"
      storage_container_name: "<container name>"