Use Edge Delta to Ingest from an OTLP Source

Prepare to consume OTLP traffic.

5 minute read

Overview

The OTLP source node consumes data items directly from OTLP configured data sources. The node is configured with the port that the agent should listen on.

Configure OTLP

To configure the OTLP source node, you must obtain the port number from the OTLP configuration:

Instrumentation Libraries: When using the OpenTelemetry SDKs, the port used to emit OTLP logs is part of the exporter configuration. The endpoint (which includes the host and port) is set when setting up the OpenTelemetry exporter within your application code. See Instrument Code using OpenTelemetry.
OpenTelemetry Collector: The port number on which the collector should send outgoing OTLP data is specified in the exporter section.
Zero-Code Instrumentation Agents: Similar to the instrumentation libraries, auto-instrumentation agents are configured to send data to a specified endpoint. This configuration includes the port number to which OTLP logs will be sent. See Instrument Code using OpenTelemetry.
Sidecars: In a Kubernetes environment, a sidecar that runs an instance of the OpenTelemetry Collector is set up using a configuration file, in which you can find the port for the OTLP receiver and exporter.
Log Routers and Forwarders: Log routers and forwarders may have plugins or output configurations that support OTLP. Within these configurations,the endpoint is defined, including the port, where the logs should be sent in OTLP format.

Example Collector Configuration (Edge Fleet)

gRPC

In a Kubernetes Environment, you need a service open for the port that the collector will send logs, metrics, and traces to. For example, you can create a ClusterIP service to expose the OTLP port to all the agents running in the cluster.

Note: For non-kubernetes environments with collectors running in the same environment as the agent, such as on a Linux VM, the collectors will be able to communicate directly with the agent’s open OTLP ports.

apiVersion: v1
kind: Service
metadata:
  name: ed-data-supply-svc
  namespace: edgedelta
spec:
  type: ClusterIP
  selector:
    edgedelta/agent-type: processor
  ports:
    - port: 4324
      name: ed-otlp
      protocol: TCP
      targetPort: 4324

You update the Collector configuration with otlp type exporters (indicating gRPC rather than otlphttp for HTTP) that match the service you created, all for the same port you configured for the service:

    exporters:
      ...
      otlp/ed-data-supply_trace:
        endpoint: 'ed-data-supply-svc.edgedelta:4324'
        tls:
          insecure: true
      otlp/ed-data-supply_metric:
        endpoint: 'ed-data-supply-svc.edgedelta:4324'
        tls:
          insecure: true
      otlp/ed-data-supply_log:
        endpoint: 'ed-data-supply-svc.edgedelta:4324'
        tls:
          insecure: true

And you update the Collector’s Pipeline to use the new exporters:

    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          ...
          - otlp/ed-data-supply_log
          ...
        metrics:
          exporters:
          ...
          - otlp/ed-data-supply_metric
          ...
        traces:
          exporters:
          ...
          - otlp/ed-data-supply_trace
          ...

Next you configure an OTLP source node with port 4324 and the gRPC protocol.

- name: OTLP_gRPC
  type: otlp_input
  port: 4324
  protocol: grpc
  read_timeout: 1m0s

HTTP

Instead of using gRPC you can send telemetry using HTTP, this example specifies a similar service.

apiVersion: v1
kind: Service
metadata:
  name: ed-data-supply-svc
  namespace: edgedelta
spec:
  type: ClusterIP
  selector:
    edgedelta/agent-type: processor
  ports:
    - port: 4324
      name: ed-otlp-http
      protocol: TCP
      targetPort: 4324

You configure otlphttp type exporters (rather than otlp for gRPC) on the port you configured for the service and disable compression.

exporters:
    otlphttp/ed-data-supply_trace:
    endpoint: "http://ed-data-supply-svc.edgedelta:4324"
    compression: none
    tls:
      insecure: true
  otlphttp/ed-data-supply_metric:
    endpoint: "http://ed-data-supply-svc.edgedelta:4324"
    compression: none
    tls:
      insecure: true
  otlphttp/ed-data-supply_log:
    endpoint: "http://ed-data-supply-svc.edgedelta:4324"
    compression: none
    tls:
      insecure: true

You include http:// and the port number at the end but no route is required.

Next, you update the Collector’s Pipeline to use the new exporters:

service:
  extensions:
    - health_check
  pipelines:
    logs:
      exporters:
        - otlphttp/ed-data-supply_log
    metrics:
      exporters:
        - otlphttp/ed-data-supply_metric
    traces:
      exporters:
        - otlphttp/ed-data-supply_trace

Finally, you configure an OTLP source node with port 4324 and the http protocol.

- name: OTLP HTTP
  type: otlp_input
  port: 4324
  protocol: http
  read_timeout: 1m0s

Example Collector Configuration (Cloud Fleet)

gRPC (Cloud Fleet)

If you are sending OTEL telemetry from the collector to a cloud fleet, you update the Collector configuration with exporters pointing to the cloud fleet. You use secure TLS and port 443:

    exporters:
      otlp/ed-data-supply_trace: 
        endpoint: '12345678-1a2b-3c4d-5e6f-7890ghijklmn-grpc-us-west2-cf.aws.edgedelta.com:443'  
        tls:
          insecure: false
      otlp/ed-data-supply_metric: 
        endpoint: '12345678-1a2b-3c4d-5e6f-7890ghijklmn-grpc-us-west2-cf.aws.edgedelta.com:443'
        tls:
          insecure: false
      otlp/ed-data-supply_log: 
        endpoint: '12345678-1a2b-3c4d-5e6f-7890ghijklmn-grpc-us-west2-cf.aws.edgedelta.com:443'
        tls:
          insecure: false

Replace the endpoint with one provided in your Cloud Fleet Overview. Include the port number but no route is required. Do not include grpcs://.

And you update the Collector’s Pipeline to use the new exporters:

    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          ...
          - otlp/ed-data-supply_log
          ...
        metrics:
          exporters:
          ...
          - otlp/ed-data-supply_metric
          ...
        traces:
          exporters:
          ...
          - otlp/ed-data-supply_trace
          ...

The Cloud Fleet contains an OTLP input node by default, which does not need to be adjusted for this gRPC configuration:

- name: otlp_input
  type: otlp_input
  port: 4317
  protocol: grpc

HTTP (Cloud Fleet)

To send OTLP telemetry to an Edge Delta Cloud Fleet you configure otlphttp exporters and disable compression. You use secure TLS and port 443 for HTTPS:

    exporters:
      otlphttp/ed-data-supply_trace:   
        endpoint: 'https://12345678-1a2b-3c4d-5e6f-7890ghijklmn-http-us-west2-cf.aws.edgedelta.com:443'  
        compression: none
        tls:
          insecure: false
      otlphttp/ed-data-supply_metric:   
        endpoint: 'https://12345678-1a2b-3c4d-5e6f-7890ghijklmn-http-us-west2-cf.aws.edgedelta.com:443' 
        compression: none
        tls:
          insecure: false    
      otlphttp/ed-data-supply_log:    
        endpoint: 'https://12345678-1a2b-3c4d-5e6f-7890ghijklmn-http-us-west2-cf.aws.edgedelta.com:443'  
        compression: none
        tls:
          insecure: false

Replace the endpoint with one provided in your Cloud Fleet Overview. You include the port number at the end but no route is required. Unlike gRPC you include https://.

And you update the Collector’s Pipeline to use the new exporters:

    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          ...
          - otlphttp/ed-data-supply_log
          ...
        metrics:
          exporters:
          ...
          - otlphttp/ed-data-supply_metric
          ...
        traces:
          exporters:
          ...
          - otlphttp/ed-data-supply_trace
          ...

The Cloud Fleet contains an HTTP input node by default, which you need to delete. Replace it with an OTLP input node listening on port 80 for HTTP traffic:

- name: otlp_input_80
  type: otlp_input
  port: 80
  protocol: http
- name: otlp_input
  type: otlp_input
  port: 4317
  protocol: grpc

Note: You may also need to include an unused gRPC OTLP node to pass configuration validation.