Edge Delta OTLP Source

Overview

The OTLP source node consumes logs directly from OTLP configured data sources. The node is configured with the port that the agent should listen on.

Note: The node can ingest either logs, metrics (processed as gauges) or traces per endpoint. To ingest logs and metrics, for example, two OTLP source nodes are required with separate endpoints.

• outgoing_data_types: log, metric, trace

Configure OTLP

Obtain the port number from the OTLP configuration:

• Instrumentation Libraries: When using the OpenTelemetry SDKs, the port used to emit OTLP logs is part of the exporter configuration. The endpoint (which includes the host and port) is set when setting up the OpenTelemetry exporter within your application code.
• OpenTelemetry Collector: The port number on which the collector should send outgoing OTLP data is specified in the exporter section.
• Auto-Instrumentation Agents: Similar to the instrumentation libraries, auto-instrumentation agents are configured to send data to a specified endpoint. This configuration includes the port number to which OTLP logs will be sent.
• Sidecars: In a Kubernetes environment, a sidecar that runs an instance of the OpenTelemetry Collector is set up using a configuration file, in which you can find the port for the OTLP receiver and exporter.
• Log Routers and Forwarders: Log routers and forwarders may have plugins or output configurations that support OTLP. Within these configurations,the endpoint is defined, including the port, where the logs should be sent in OTLP format.

Example Collector Configuration

In a Kubernetes Environment, you need a service open for the ports that the collector will send logs, metrics, and traces to. For example, you can create a ClusterIP service to expose the OTLP ports to all the agents running in the cluster.

Note: For non-kubernetes environments with collectors running in the same environment as the agent, such as on a Linux VM, the collectors will be able to communicate directly with the agent’s open OTLP ports.

apiVersion: v1
kind: Service
metadata:
  name: ed-data-supply-svc
  namespace: edgedelta
spec:
  type: ClusterIP
  selector:
    edgedelta/agent-type: processor
  ports:
    - port: 4324
      name: ed-otlp-logs
      protocol: TCP
      targetPort: 4324
    - port: 4325
      name: ed-otlp-metrics
      protocol: TCP
      targetPort: 4325
    - port: 4326
      name: ed-otlp-traces
      protocol: TCP
      targetPort: 4326

You update the Collector configuration with exporters that match the service you created:

    exporters:
      ...
      otlp/ed-data-supply_trace:
        endpoint: 'ed-data-supply-svc.edgedelta:4326'
        tls:
          insecure: true
      otlp/ed-data-supply_metric:
        endpoint: 'ed-data-supply-svc.edgedelta:4325'
        tls:
          insecure: true
      otlp/ed-data-supply_log:
        endpoint: 'ed-data-supply-svc.edgedelta:4324'
        tls:
          insecure: true

Note: gRPC traffic and HTTP are supported by the OTLP source node. Therefore the OTLP type must be set to otlp or otlphttp depending on the type. You also need to specify the protocol parameter.

And you update the Collector’s Pipeline to use the new exporters:

    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          ...
          - otlp/ed-data-supply_log
          ...
        metrics:
          exporters:
          ...
          - otlp/ed-data-supply_metric
          ...
        traces:
          exporters:
          ...
          - otlp/ed-data-supply_trace
          ...

Example Configuration

Create separate OTLP sources for each port to ingest logs, metrics and traces from the collector.

nodes:
- name: OTLP traces
  type: otlp_input
  port: 4326
  protocol: grpc
  read_timeout: 1m0s
- name: OTLP logs
  type: otlp_input
  port: 4324
  protocol: grpc
  read_timeout: 1m0s
- name: OTLP metrics
  type: otlp_input
  port: 4325
  protocol: grpc
  read_timeout: 1m0s

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: otlp_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

port

The port parameter specifies the port number to listen on. It is specified as an integer and is required.

nodes:
- name: <node name>
  type: otlp_input
  port: <port number>

protocol

The protocol parameter defines which protocol is being used to send items to the OTLP source. You can specify either grpc or http. If you select HTTP you can also specify a read timeout. The default is gRPC and a protocol is required.

nodes:
- name: <node name>
  type: otlp_input
  port: <port number>
  protocol: http|grpc

Optional Parameters

read_timeout

The read_timeout parameter defines a wait time timeout for HTTP connections. You specify it a duration. The default is 1m and it is optional.

nodes:
- name: <node name>
  type: otlp_input
  port: <port number>
  protocol: http
  read_timeout: 2m