Azure Function Monitoring

Overview

You can deploy Edge Delta components to an AKS cluster.

Azure functions generate various telemetry data, such as:

• Traces
• Error
• Exceptions
• RemoteDependencies
• Requests

This telemetry data is ingested directly into the application insight from Azure functions.

To learn more about Azure’s built-in sampling techniques, review this document from Microsoft.

Compared to Azure, Edge Delta provides smarter sampling options, such as collecting telemetry data specifically from failed functions and from a small percentage of successful ones. With Edge Delta, application insight telemetry ingestion costs can be reduced drastically.

Step 1: Set Up the Edge Delta Processor in an AKS Cluster

Before you begin, you must have your app insight’s instrument key available.

1. In the Edge Delta App click Data Pipeline, and then click Agent Configs.
2. Create or update a configuration:

• To create a new configuration, click Create Configuration, and create a Kubernetes configuration.
• To update an existing configuration, locate the desired configuration, then under Actions , click the vertical ellipses, and then click Edit.

3. To populate the YAML file, copy and paste the content from this file.
4. In the YAML file, replace INSTRUMENTATION_KEY with your the instrumentation key.
5. Click Save.
6. Create a new AKS cluster or use an existing cluster.
7. Add a new node pool on AKS with the following specs:

  name: processors
    OS: linux
    size: 1
    SKU: Standard_D4s_v3

If you skip this step, then update the nodeSelector in ed-appinsights-trace-processor.yaml. 8. When cluster is ready, connect to the cluster, and then create the Edge Delta Secret API key:

kubectl create namespace edgedelta
kubectl create secret generic ed-api-key \
    --namespace=edgedelta \
    --from-literal=ed-api-key="c40bafd5-xxxxxxx"

9. Create the ingress resources described in this document from Microsoft. This step requires Helm.

kubectl create namespace ingress-basic
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install nginx-ingress ingress-nginx/ingress-nginx \
    --namespace ingress-basic \
    --set controller.replicaCount=1 \
    --set controller.nodeSelector.agentpool=processors \
    --set defaultBackend.nodeSelector.agentpool=processors

10. Get the IP address:

kubectl --namespace ingress-basic get services -o wide -w nginx-ingress-ingress-nginx-controller

11. Create a DNS zone on the Azure portal, based on this document from Microsoft. You will need to have a public DNS entry for your zone to be publicly accessible. For example, if your DNS zone is contoso.xyz, then you need to own contoso.xyz. As a workaround, you can create a separate AKS cluster with http application routing enabled and use that DNS zone. The DNS zone will have public DNS records created by Azure.
12. Create an A record in your DNS zone which points to the IP address of ingress controller:

ingest.edgedelta.<your dns zone> -> <IP Address from above step>

13. Install cert-manager:

kubectl label namespace ingress-basic cert-manager.io/disable-validation=true
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
    cert-manager \
    --namespace ingress-basic \
    --version v0.16.1 \
    --set installCRDs=true \
    --set nodeSelector."beta\.kubernetes\.io/os"=linux \
    jetstack/cert-manager

14. With the DNS entry you configured earlier, update the host values in the edgedelta-ingress resource, based on ed-appinsights-trace-processor.yaml.
15. Create the Edge Delta http recorder and agent:

kubectl apply -f ed-appinsights-trace-processor.yaml

kubectl get pods -n edgedelta

17. Verify certificate creation:

kubectl get certificate --namespace ingress-basic

18. Verify the public endpoint:

https://ingest.edgedelta.198de54f02b345ab92a8.centralus.aksapp.io/

Step 2: Set Up Azure Function With Dual Telemetry Write Mode

1. Navigate to the Azure application folder, and then add dependencies:

dotnet add package Microsoft.Azure.Functions.Extensions
dotnet add package Microsoft.Extensions.Logging.ApplicationInsights

2. Create a StartUp.cs file under the targeted Azure function application with the content in this StartUp.cs file. Update the namespace at the top. In the example below, the custom sinker implementation called ForkingTelemetryChannel replicates telemetry data for ingestion into a secondary ingestion endpoint. Note that the dual ingestion process is parallelized to reduce the overall latency.

...
public void Send(ITelemetry item)
{
    var itemDup = item.DeepClone();
    itemDup.Context.InstrumentationKey = this.secondaryInstrumentationKey;
    Parallel.Invoke(
        () => { orginalChannel.Send(item); },
        () => { secondaryChannel.Send(itemDup); }
    );
}
...

3. Set the secondary application insight connection string. Provide the secondary endpoint address created in the AKS cluster ingress endpoint. Provide the secondary instrumentation key that will be used to forward the matching telemetry data to the application insight from the Edge Delta processor.

{
...
  "Values": {
...
    // Either original instrumentation key or original connection string is provided but not both
    "APPINSIGHTS_INSTRUMENTATIONKEY": "OriginalKey123";
    "APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=OriginalKey123;IngestionEndpoint=https://dc.services.visualstudio.com",
    // Required secondary connection string for forking application insight traffic
    "APPLICATIONINSIGHTS_SECONDARY_CONNECTION_STRING": "InstrumentationKey=SecondaryKey123;IngestionEndpoint=https://ingest.edgedelta.198de54f02b345ab92a8.centralus.aksapp.io",
...
  },
...
}

Step 3: Set Up Azure Function Without Dual Writes

Set the application insight connection string to point to the public endpoint previously. Use your target appinsight instrumentation key.

"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=***;IngestionEndpoint=https://ingest.edgedelta.198de54f02b345ab92a8.centralus.aksapp.io",

Step 4: Test the Setup

1. Run the Azure functions and simulate failure scenarios.
2. Visit the application insights to verify that the Edge Delta agent is forwarding failed traces.