Configuring Azure to Stream to Event Hub

Overview

Before Edge Delta can consume data from Azure Event Hub, you need to configure Azure services to stream their logs, metrics, and telemetry to the Event Hub. This guide covers the most common Azure streaming scenarios and provides validation steps to ensure data is flowing correctly.

Time estimate: 15-20 minutes (assuming Azure resources are already created)

Before You Begin

Ensure you have completed these prerequisites:

• Active Azure subscription with appropriate permissions
• Azure Event Hub namespace and Event Hub created (Create Event Hub resources)
• Access to configure diagnostic settings for Azure resources
• PowerShell 7+ or Azure CLI installed for validation
• Basic understanding of Azure Monitor and diagnostic settings

Azure Event Hub Setup

Prerequisites: Create Event Hub Resources

Before configuring streaming, you need an Event Hub namespace and Event Hub. You can set up these resources using our automated template or manually through Azure Portal.

Option A: Automated Setup (Recommended)

Deploy all required Event Hub infrastructure with a single click using our ARM template. This creates everything Edge Delta needs: Event Hub namespace, Event Hub, access policies, storage account, and checkpoint container.

Interactive deployment wizard (v1.0.12+):

The deployment wizard provides an interactive experience with dropdowns to select existing resources:

1. Select existing resource group - Choose from your resource groups (location inherited automatically)
2. Choose deployment mode:
   • Create all new resources - Event Hub namespace + storage account
   • Use existing namespace - Create Event Hub in your existing namespace
   • Use existing storage - Create checkpoint container in your existing storage account
   • Use both existing - Maximum flexibility with existing infrastructure

Always creates:

• Event Hub named edgedelta-logs (or custom name)
• Consumer group: edgedelta-processors
• Access policies: AzureSendPolicy (Send) and EdgeDeltaListenPolicy (Listen)
• Blob container: edgedelta-checkpoints for checkpointing

Optionally creates (based on your selection):

• Event Hub namespace (Standard tier, TLS 1.2+ security, configurable partitions)
• Storage account for Edge Delta checkpointing
• Subscription activity log diagnostic settings (manual configuration recommended)

Prerequisites:

• Existing resource group (must be created before deployment)
• Standard or Premium tier Event Hub namespace (if using existing - Basic tier not supported)

Estimated cost: $11-25/month (Standard tier, varies by region and throughput)

See the ARM template documentation for detailed deployment instructions, parameters, and manual deployment options.

Option B: Manual Setup

If you prefer manual configuration, create these resources following Microsoft’s documentation:

Required Azure resources:

1. Event Hub Namespace - Create following Microsoft’s namespace setup guide
2. Event Hub - Create within the namespace
3. Shared Access Policy - Create with Send permission (for Azure services to write) and Listen permission (for Edge Delta to read)
4. Storage Account - Required for Edge Delta checkpointing (Create storage account)

Configuration notes for Edge Delta:

• Recommended tier: Standard or Premium (Basic tier has limitations)
• Minimum partitions: 2-4 for optimal throughput
• Consumer group: Create a dedicated group named edgedelta-processors (do not use $Default)
• Message retention: 1-7 days depending on data volume and processing latency requirements

Validate Event Hub Connectivity

Verify your Event Hub is operational before configuring streaming:

PowerShell:

# Install Azure PowerShell module if needed
# Install-Module -Name Az -AllowClobber -Force

# Login to Azure
Connect-AzAccount

# Set your subscription context
$subscriptionId = "your-subscription-id"
Set-AzContext -SubscriptionId $subscriptionId

# Verify Event Hub exists
$resourceGroup = "your-resource-group"
$namespace = "your-eventhub-namespace"
$eventHubName = "your-eventhub-name"

Get-AzEventHub -ResourceGroupName $resourceGroup `
               -NamespaceName $namespace `
               -EventHubName $eventHubName

# Expected output: Event Hub details including PartitionCount, Status

Azure CLI:

# Login to Azure
az login

# Set subscription
az account set --subscription "your-subscription-id"

# Verify Event Hub exists
az eventhubs eventhub show \
  --resource-group your-resource-group \
  --namespace-name your-eventhub-namespace \
  --name your-eventhub-name

# Expected output: JSON with Event Hub configuration

Configure Azure Services to Stream Logs

Azure provides multiple methods to stream telemetry data to Event Hub. Choose the appropriate method based on your data source.

Option 1: Resource Diagnostic Settings (Most Common)

Stream logs from specific Azure resources (App Services, VMs, SQL Databases, Function Apps, etc.) to Event Hub.

Configuration Steps

1. Navigate to your Azure resource in the Azure Portal (e.g., App Service, Virtual Machine, SQL Database)

2. In the left menu under Monitoring, select Diagnostic settings

3. Click + Add diagnostic setting

4. Configure the diagnostic setting:

   • Diagnostic setting name: Enter a descriptive name (e.g., Stream-to-EdgeDelta-EventHub)
   • Logs: Check the log categories relevant to your monitoring needs
   • Metrics: (Optional) Check AllMetrics for performance data

5. Under Destination details:

   • Check Stream to an event hub
   • Subscription: Select your Azure subscription
   • Event hub namespace: Select your Event Hub namespace
   • Event hub name: Select your Event Hub (or leave as “insights-logs-*” for auto-generated names)
   • Event hub policy name: Select a policy with Send permissions

6. Click Save

Validate Resource Diagnostic Streaming

PowerShell:

# Get diagnostic settings for a resource
$resourceId = "/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Web/sites/{app-service-name}"

Get-AzDiagnosticSetting -ResourceId $resourceId

# Expected output: List of diagnostic settings with EventHub configuration

Generate Test Activity

To verify logs are flowing, generate activity on your Azure resource:

For App Services:

# Make HTTP requests to your App Service
$appServiceUrl = "https://your-app-service.azurewebsites.net"
Invoke-WebRequest -Uri $appServiceUrl

# Wait 5-10 minutes, then check Event Hub metrics

For SQL Database:

# Execute a test query (generates QueryStoreRuntimeStatistics logs)
# Connect to your SQL Database and run any query

Option 2: Subscription Activity Logs

Capture subscription-level activity showing who performed what actions across all Azure resources.

Configuration Steps

1. In Azure Portal, navigate to Azure Monitor

2. Select Activity log from the left menu

3. Click Export Activity Logs (or Diagnostic settings button)

4. Click + Add diagnostic setting

5. Configure the setting:

   • Diagnostic setting name: Activity-to-EventHub
   • Categories: Select the activity types to capture:
     • Administrative (resource creation, modification, deletion)
     • Security (security-related events)
     • Service Health (Azure service issues)
     • Alert (alert activations)
     • Recommendation (Azure Advisor recommendations)
     • Policy (Azure Policy evaluations)
     • Autoscale (autoscaling operations)
     • Resource Health (resource health changes)

6. Under Destination details:

   • Check Stream to an event hub
   • Select your Event Hub namespace and Event Hub

7. Click Save

Validate Activity Log Streaming

PowerShell:

# Get activity log diagnostic settings
$subscriptionId = (Get-AzContext).Subscription.Id
$resourceId = "/subscriptions/$subscriptionId"

Get-AzDiagnosticSetting -ResourceId $resourceId

# Expected output: Activity log diagnostic settings with EventHub

Generate Test Activity

Create a test resource to generate activity logs:

PowerShell:

# Create a test resource group (generates Administrative activity)
$testRgName = "test-activity-log-$(Get-Random)"
$location = "eastus"

New-AzResourceGroup -Name $testRgName -Location $location

# Wait 2-5 minutes, then check Event Hub metrics

# Clean up test resource
Remove-AzResourceGroup -Name $testRgName -Force

Option 3: Application Insights Continuous Export

Stream application telemetry (requests, exceptions, traces, metrics) from Application Insights to Event Hub.

Configuration Steps

Using Diagnostic Settings (Recommended):

1. Navigate to your Application Insights resource

2. Under Monitoring, select Diagnostic settings

3. Click + Add diagnostic setting

4. Select telemetry categories:

   • AppAvailabilityResults
   • AppBrowserTimings
   • AppEvents
   • AppMetrics
   • AppDependencies
   • AppExceptions
   • AppPageViews
   • AppPerformanceCounters
   • AppRequests
   • AppSystemEvents
   • AppTraces

5. Check Stream to an event hub and configure Event Hub details

6. Click Save

Validate Application Insights Streaming

PowerShell:

# Get Application Insights diagnostic settings
$aiResourceId = "/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.Insights/components/{app-insights-name}"

Get-AzDiagnosticSetting -ResourceId $aiResourceId

# Expected output: Diagnostic settings with telemetry categories

Option 4: Custom Applications (Programmatic)

For custom applications, use Azure Event Hubs SDKs to send data programmatically.

Azure Event Hubs SDK Example

C# (.NET):

using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;

var connectionString = "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=your-policy;SharedAccessKey=your-key;EntityPath=your-eventhub";

await using var producer = new EventHubProducerClient(connectionString);

// Create a batch
using var eventBatch = await producer.CreateBatchAsync();

// Add events to batch
var logData = new
{
    timestamp = DateTime.UtcNow,
    level = "INFO",
    message = "Custom application log",
    service = "my-service"
};

var eventData = new EventData(System.Text.Encoding.UTF8.GetBytes(
    System.Text.Json.JsonSerializer.Serialize(logData)
));

if (!eventBatch.TryAdd(eventData))
{
    throw new Exception("Event too large for batch");
}

// Send the batch
await producer.SendAsync(eventBatch);

PowerShell:

# Send test event using Azure CLI
$eventHubName = "your-eventhub-name"
$namespace = "your-namespace"
$resourceGroup = "your-resource-group"

# Create test event data
$eventData = @{
    timestamp = (Get-Date).ToUniversalTime().ToString("o")
    level = "INFO"
    message = "Test event from PowerShell"
} | ConvertTo-Json

# Send event (requires Azure CLI with Event Hubs extension)
az eventhubs eventhub send `
  --resource-group $resourceGroup `
  --namespace-name $namespace `
  --name $eventHubName `
  --body $eventData

Python:

from azure.eventhub import EventHubProducerClient, EventData
import json
from datetime import datetime

connection_str = "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=your-policy;SharedAccessKey=your-key;EntityPath=your-eventhub"

producer = EventHubProducerClient.from_connection_string(connection_str)

event_data = {
    "timestamp": datetime.utcnow().isoformat(),
    "level": "INFO",
    "message": "Custom log from Python",
    "service": "my-service"
}

event_data_batch = producer.create_batch()
event_data_batch.add(EventData(json.dumps(event_data)))

producer.send_batch(event_data_batch)
producer.close()

Verify Events Are Flowing to Event Hub

After configuring streaming, validate that events are reaching your Event Hub.

Check Event Hub Metrics

Azure Portal:

1. Navigate to your Event Hub Namespace
2. Select your Event Hub
3. Click Metrics in the left menu
4. Add these metrics:
   • Incoming Messages - Should show non-zero values if logs are flowing
   • Incoming Bytes - Data volume being received
   • Outgoing Messages - Messages consumed (will be zero until Edge Delta connects)

PowerShell:

# Get Event Hub metrics for the last hour
$namespace = "your-eventhub-namespace"
$eventHubName = "your-eventhub-name"
$resourceGroup = "your-resource-group"

# Get the Event Hub resource ID
$eventHubId = (Get-AzEventHub -ResourceGroupName $resourceGroup `
                               -NamespaceName $namespace `
                               -EventHubName $eventHubName).Id

# Query incoming message metrics
$endTime = Get-Date
$startTime = $endTime.AddHours(-1)

Get-AzMetric -ResourceId $eventHubId `
             -MetricName "IncomingMessages" `
             -StartTime $startTime `
             -EndTime $endTime `
             -TimeGrain 00:05:00 `
             -AggregationType Total

# Expected output: Metric values showing message counts
# If all values are 0, events are not reaching Event Hub

View Event Hub Data (Portal)

Azure Portal Data Explorer:

1. Navigate to your Event Hub
2. Select Process Data from left menu
3. Click Explore under “Get started with sample queries”
4. You should see sample events if data is flowing

Check for Configuration Issues

PowerShell Diagnostic Script:

# Comprehensive Event Hub validation script

param(
    [Parameter(Mandatory=$true)]
    [string]$ResourceGroupName,

    [Parameter(Mandatory=$true)]
    [string]$NamespaceName,

    [Parameter(Mandatory=$true)]
    [string]$EventHubName
)

Write-Host "=== Event Hub Validation ===" -ForegroundColor Cyan

# 1. Check Event Hub exists
Write-Host "`nChecking Event Hub existence..." -ForegroundColor Yellow
try {
    $eventHub = Get-AzEventHub -ResourceGroupName $ResourceGroupName `
                                -NamespaceName $NamespaceName `
                                -EventHubName $EventHubName
    Write-Host "✓ Event Hub found: $($eventHub.Name)" -ForegroundColor Green
    Write-Host "  Partitions: $($eventHub.PartitionCount)" -ForegroundColor Gray
    Write-Host "  Status: $($eventHub.Status)" -ForegroundColor Gray
} catch {
    Write-Host "✗ Event Hub not found or access denied" -ForegroundColor Red
    exit 1
}

# 2. Check Shared Access Policies
Write-Host "`nChecking Shared Access Policies..." -ForegroundColor Yellow
$policies = Get-AzEventHubAuthorizationRule -ResourceGroupName $ResourceGroupName `
                                             -NamespaceName $NamespaceName `
                                             -EventHubName $EventHubName

foreach ($policy in $policies) {
    $rights = ($policy.Rights -join ", ")
    Write-Host "  Policy: $($policy.Name) - Rights: $rights" -ForegroundColor Gray
}

if ($policies.Count -eq 0) {
    Write-Host "✗ No access policies found - Azure services cannot send data" -ForegroundColor Red
} else {
    Write-Host "✓ Found $($policies.Count) access policy/policies" -ForegroundColor Green
}

# 3. Check Consumer Groups
Write-Host "`nChecking Consumer Groups..." -ForegroundColor Yellow
$consumerGroups = Get-AzEventHubConsumerGroup -ResourceGroupName $ResourceGroupName `
                                               -NamespaceName $NamespaceName `
                                               -EventHubName $EventHubName

foreach ($group in $consumerGroups) {
    Write-Host "  Consumer Group: $($group.Name)" -ForegroundColor Gray
}

if ($consumerGroups.Name -notcontains "edgedelta-processors") {
    Write-Host "⚠ Recommended consumer group 'edgedelta-processors' not found" -ForegroundColor Yellow
    Write-Host "  Consider creating: New-AzEventHubConsumerGroup -ResourceGroupName $ResourceGroupName -NamespaceName $NamespaceName -EventHubName $EventHubName -Name 'edgedelta-processors'" -ForegroundColor Gray
}

# 4. Check recent metrics
Write-Host "`nChecking recent activity (last hour)..." -ForegroundColor Yellow
$endTime = Get-Date
$startTime = $endTime.AddHours(-1)

$metrics = Get-AzMetric -ResourceId $eventHub.Id `
                        -MetricName "IncomingMessages" `
                        -StartTime $startTime `
                        -EndTime $endTime `
                        -TimeGrain 00:05:00 `
                        -AggregationType Total

$totalMessages = ($metrics.Data.Total | Measure-Object -Sum).Sum

if ($totalMessages -gt 0) {
    Write-Host "✓ Incoming messages detected: $totalMessages messages in last hour" -ForegroundColor Green
} else {
    Write-Host "✗ No incoming messages in last hour" -ForegroundColor Red
    Write-Host "  Possible causes:" -ForegroundColor Yellow
    Write-Host "  - Diagnostic settings not configured" -ForegroundColor Gray
    Write-Host "  - No activity to log" -ForegroundColor Gray
    Write-Host "  - Permissions issue with sending policy" -ForegroundColor Gray
}

Write-Host "`n=== Validation Complete ===" -ForegroundColor Cyan

Save this script as Test-EventHubSetup.ps1 and run:

.\Test-EventHubSetup.ps1 -ResourceGroupName "your-rg" `
                         -NamespaceName "your-namespace" `
                         -EventHubName "your-eventhub"

Common Issues and Solutions

No Events Appearing in Event Hub

Symptoms: Event Hub metrics show zero incoming messages after 15+ minutes

Solutions:

1. Verify diagnostic settings are active:

   # Check if diagnostic setting is enabled
   $resourceId = "/subscriptions/{sub-id}/resourceGroups/{rg}/providers/{provider}/{resource}"
   $settings = Get-AzDiagnosticSetting -ResourceId $resourceId
   $settings | Format-List Name, EventHubName, EventHubAuthorizationRuleId
   

2. Check access policy permissions:

   • Policy must have Send permission
   • Verify policy is correctly selected in diagnostic settings

3. Generate test activity:

   • For resource logs: Use the resource (make HTTP requests, run queries, etc.)
   • For activity logs: Create/modify/delete a test resource
   • Some resources require specific activity types to generate logs

4. Check Event Hub namespace status:

   Get-AzEventHubNamespace -ResourceGroupName $resourceGroup -NamespaceName $namespace
   # Status should be "Active"
   

Diagnostic Setting Save Fails

Symptoms: Error when saving diagnostic settings: “The specified policy does not have send claims”

Solution:

# Create or update authorization rule with Send permission
New-AzEventHubAuthorizationRule -ResourceGroupName $resourceGroup `
                                 -NamespaceName $namespace `
                                 -EventHubName $eventHubName `
                                 -Name "DiagnosticSettingsPolicy" `
                                 -Rights @("Send")

# Get the rule ID for diagnostic settings
$rule = Get-AzEventHubAuthorizationRule -ResourceGroupName $resourceGroup `
                                         -NamespaceName $namespace `
                                         -EventHubName $eventHubName `
                                         -Name "DiagnosticSettingsPolicy"
Write-Host "Use this Rule ID: $($rule.Id)"

Events Delayed or Intermittent

Symptoms: Events appear sporadically or with significant delay

Causes and solutions:

1. Partition throttling:

   • Increase partition count: 2-4 for low volume, 8-32 for high volume
   • Upgrade Event Hub tier (Standard supports more partitions than Basic)

2. Throughput units exhausted:

   # Check throughput usage
   Get-AzEventHubNamespace -ResourceGroupName $resourceGroup `
                            -NamespaceName $namespace |
       Select-Object Name, MaximumThroughputUnits, @{Name="CurrentThroughputUnits";Expression={$_.Sku.Capacity}}
   

3. Enable auto-inflate:

   Set-AzEventHubNamespace -ResourceGroupName $resourceGroup `
                            -NamespaceName $namespace `
                            -EnableAutoInflate $true `
                            -MaximumThroughputUnits 20
   

Wrong Event Hub Receiving Data

Symptoms: Events going to unexpected Event Hub or auto-generated Event Hub name

Solution:

# List all Event Hubs in namespace to find where data is going
Get-AzEventHub -ResourceGroupName $resourceGroup -NamespaceName $namespace |
    Select-Object Name, PartitionCount

# Update diagnostic setting to use specific Event Hub
Set-AzDiagnosticSetting -ResourceId $resourceId `
                         -Name "your-diagnostic-setting" `
                         -EventHubName "your-specific-eventhub" `
                         -EventHubAuthorizationRuleId $authRuleId

Next Steps

Once you’ve verified that events are flowing to your Event Hub:

1. Configure Edge Delta: Return to the Azure Event Hub Source documentation to configure Edge Delta to consume from your Event Hub

2. Create dedicated consumer group:

   New-AzEventHubConsumerGroup -ResourceGroupName $resourceGroup `
                                -NamespaceName $namespace `
                                -EventHubName $eventHubName `
                                -Name "edgedelta-processors"
   

3. Retrieve connection string for Edge Delta:

   # Get connection string with Listen permission for Edge Delta
   $listenRule = Get-AzEventHubAuthorizationRule -ResourceGroupName $resourceGroup `
                                                  -NamespaceName $namespace `
                                                  -EventHubName $eventHubName `
                                                  -Name "EdgeDeltaListenPolicy"
   
   $keys = Get-AzEventHubKey -ResourceGroupName $resourceGroup `
                              -NamespaceName $namespace `
                              -EventHubName $eventHubName `
                              -Name "EdgeDeltaListenPolicy"
   
   Write-Host "Connection String (copy for Edge Delta configuration):"
   Write-Host $keys.PrimaryConnectionString -ForegroundColor Green
   

4. Set up checkpoint storage:

   • Create Azure Storage account if not already exists
   • Create blob container for checkpoints
   • Retrieve storage account key for Edge Delta configuration

Additional Resources

• Azure Event Hubs documentation
• Diagnostic settings in Azure Monitor
• Azure Activity log
• Application Insights telemetry
• Event Hubs quotas and limits