Troubleshooting OTLP Destination

Troubleshoot Edge Delta OTLP destination nodes including connection issues, authentication failures, and configuration problems.

7 minute read

Overview

This guide provides comprehensive troubleshooting for the Edge Delta OTLP Destination node when sending telemetry data (logs, metrics, and traces) to OTLP-compatible endpoints using gRPC or HTTP protocols.

Note: This page covers troubleshooting the OTLP destination node (sending data FROM Edge Delta). For troubleshooting OTLP data sent TO Edge Delta Cloud Pipelines, see Troubleshooting Cloud Pipelines.

Quick Diagnostic Checklist

Before diving into specific issues, verify:

Correct protocol selected (gRPC or HTTP)
Endpoint format is correct (no http:// or https:// prefix)
Port number matches the service (typically 443 for TLS, 4317 for gRPC, 4318 for HTTP)
TLS is enabled when required (tls.enabled: true for HTTPS endpoints)
Encoding is specified for HTTP protocol (proto or json)
Authentication headers are configured correctly
Network connectivity to the OTLP receiver

Common Error Messages

Error: “context deadline exceeded”

Symptoms

OTLP-Streamer-otlp_output_200e: Post "http://endpoint.example.com:443/v1/logs": context deadline exceeded

Causes and Solutions

1. Incorrect Protocol in Endpoint

The endpoint field should NOT include the protocol prefix (http:// or https://). Edge Delta automatically constructs the full URL based on your configuration.

Incorrect:

nodes:
  - name: my_otlp
    type: otlp_output
    endpoint: https://endpoint.example.com/receiver/v1/otlp
    port: 443

Correct:

nodes:
  - name: my_otlp
    type: otlp_output
    endpoint: endpoint.example.com/receiver/v1/otlp
    port: 443
    protocol: http
    tls:
      enabled: true

2. Wrong Port Number

Ensure the port matches your endpoint’s expected port:

gRPC without TLS: typically port 4317
HTTP without TLS: typically port 4318
gRPC with TLS: typically port 443
HTTP with TLS: typically port 443

3. Missing TLS Configuration

If your endpoint uses HTTPS (port 443), you must enable TLS:

nodes:
  - name: my_otlp
    type: otlp_output
    endpoint: endpoint.example.com/receiver
    port: 443
    protocol: http
    tls:
      enabled: true

4. Network Connectivity Issues

Test connectivity to the endpoint:

# For HTTP endpoints
curl -v https://endpoint.example.com:443/v1/logs

# For gRPC endpoints
grpcurl -v endpoint.example.com:443 list

Error: “401 Unauthorized” or “Failed to authenticate”

Symptoms

OTLP-Streamer-otlp_output_57d4: OTLP submitLogs had unexpected status code: 401 with response: Failed to authenticate

Causes and Solutions

1. Missing Authentication Headers

Many OTLP receivers require authentication. Add the appropriate headers to your configuration:

nodes:
  - name: my_otlp
    type: otlp_output
    endpoint: endpoint.example.com/receiver/v1/otlp
    port: 443
    protocol: http
    encoding: proto
    tls:
      enabled: true
    headers:
      - header: Authorization
        value: Bearer your-token-here

2. Incorrect Token Format

Different services use different authentication schemes:

Bearer Token:

headers:
  - header: Authorization
    value: Bearer YOUR_TOKEN

API Token (e.g., Dynatrace):

headers:
  - header: Authorization
    value: Api-Token YOUR_TOKEN

Embedded in Endpoint (e.g., Sumo Logic): For some services like Sumo Logic, the authentication token is part of the endpoint path:

endpoint: endpoint.collection.sumologic.com/receiver/v1/otlp/YOUR_TOKEN_HERE

3. Expired or Invalid Credentials

Verify your token or API key is:

Valid and not expired
Has the correct permissions for OTLP ingestion
Correctly copied without extra spaces or line breaks

Error: “no such host” or URL Parsing Issues

Symptoms

OTLP-Streamer-otlp_output_200e: Post "http://https//endpoint.example.com:80/v1/logs": dial tcp: lookup https on 172.20.0.10:53: no such host

Cause

The endpoint field contains https:// or http:// prefix, causing Edge Delta to construct an invalid URL.

Solution

Remove the protocol prefix from the endpoint field:

Incorrect:

endpoint: https://endpoint.example.com/receiver

Correct:

endpoint: endpoint.example.com/receiver
tls:
  enabled: true

Error: “Port appended to URI path”

Symptoms

The error response shows the port number appearing in the URI path:

URI: /receiver/v1/otlp/token:443/v1/logs

Cause

This is expected behavior when using HTTP protocol. Edge Delta constructs the URL by appending the port to the endpoint and then adds the OTLP-specific path (/v1/logs, /v1/metrics, /v1/traces).

Solution

Ensure your OTLP receiver can handle this URL format. If not, adjust your endpoint path:

If your receiver expects:

https://endpoint.example.com:443/receiver/v1/otlp/v1/logs

Configure your endpoint as:

endpoint: endpoint.example.com/receiver/v1/otlp

Edge Delta will construct: https://endpoint.example.com:443/receiver/v1/otlp/v1/logs

Protocol-Specific Troubleshooting

gRPC Protocol

Common Issues

1. Connection Refused

Test the gRPC endpoint:

grpcurl -v your-endpoint.com:4317 list

2. TLS Handshake Failures

Verify TLS configuration:

nodes:
  - name: my_otlp_grpc
    type: otlp_output
    endpoint: secure-endpoint.example.com
    port: 443
    protocol: grpc
    tls:
      enabled: true
      ignore_certificate_check: false
      ca_file: /path/to/ca.pem

3. Certificate Verification Errors

If using self-signed certificates or custom CAs:

tls:
  enabled: true
  ca_file: /path/to/custom-ca.pem

For testing only (not recommended for production):

tls:
  enabled: true
  ignore_certificate_check: true

HTTP Protocol

Common Issues

1. Missing Encoding Parameter

HTTP protocol requires the encoding parameter:

Incorrect:

protocol: http

Correct:

protocol: http
encoding: proto  # or 'json'

2. Encoding Selection

Protobuf (proto): Recommended for better performance and smaller payload size
JSON (json): Use when debugging or if the receiver requires JSON format

nodes:
  - name: my_otlp_http
    type: otlp_output
    endpoint: endpoint.example.com/receiver
    port: 443
    protocol: http
    encoding: proto

3. Compression Issues

Some OTLP receivers may not support compression. If you experience issues, compression can be disabled at the receiver level (this is typically not an Edge Delta configuration).

Destination-Specific Configuration

Dynatrace

Configuration:

nodes:
  - name: dynatrace_otlp
    type: otlp_output
    endpoint: your-env-id.live.dynatrace.com/api/v2/otlp
    port: 443
    protocol: http
    encoding: json
    tls:
      enabled: true
    headers:
      - header: Authorization
        value: Api-Token YOUR_DYNATRACE_TOKEN

Common Issues:

Ensure the token has the “Ingest OpenTelemetry traces and metrics” permission
Use encoding: json (Dynatrace prefers JSON)
Include the full API path: /api/v2/otlp

Sumo Logic

Configuration:

nodes:
  - name: sumologic_otlp
    type: otlp_output
    endpoint: endpoint.collection.sumologic.com/receiver/v1/otlp/YOUR_COLLECTOR_TOKEN
    port: 443
    protocol: http
    encoding: proto
    tls:
      enabled: true

Common Issues:

The collector token is part of the endpoint path, not a header
Must use encoding: proto (protobuf)
Ensure the endpoint includes the full path with token

Grafana Cloud

Configuration:

nodes:
  - name: grafana_otlp
    type: otlp_output
    endpoint: otlp-gateway-prod-us-east-0.grafana.net/otlp
    port: 443
    protocol: http
    encoding: proto
    tls:
      enabled: true
    headers:
      - header: Authorization
        value: Basic BASE64_ENCODED_CREDENTIALS

Common Issues:

Credentials must be base64 encoded: echo -n "instance_id:token" | base64
Use encoding: proto for better performance
Ensure the instance ID and token are correct

AWS Distro for OpenTelemetry (ADOT)

Configuration:

nodes:
  - name: adot_otlp
    type: otlp_output
    endpoint: localhost
    port: 4317
    protocol: grpc

Common Issues:

ADOT Collector typically runs as a sidecar on localhost
Default gRPC port is 4317, HTTP port is 4318
TLS is usually not required for localhost connections

Verification and Testing

Check Edge Delta Logs

View Edge Delta agent logs to see detailed error messages:

Docker:

docker logs <container-id> | grep -i otlp

Kubernetes:

kubectl logs -n edgedelta <pod-name> | grep -i otlp

Linux:

journalctl -u edgedelta | grep -i OTLP

Test Configuration

Create a minimal test configuration to isolate issues:

nodes:
  - name: test_otlp
    type: otlp_output
    endpoint: your-endpoint.com
    port: 443
    protocol: http
    encoding: proto
    tls:
      enabled: true
    headers:
      - header: Authorization
        value: Bearer test-token

workflows:
  - name: test_workflow
    source_labels:
      - test_source
    destination_labels:
      - test_otlp

Send test data and monitor logs for errors.

Network Diagnostics

Test HTTP Endpoint:

curl -X POST https://endpoint.example.com:443/v1/logs \
  -H "Content-Type: application/x-protobuf" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  --data-binary @test-payload.pb

Test gRPC Endpoint:

grpcurl -d '{}' -H "Authorization: Bearer YOUR_TOKEN" \
  endpoint.example.com:443 \
  opentelemetry.proto.collector.logs.v1.LogsService/Export

Check DNS Resolution:

nslookup endpoint.example.com
dig endpoint.example.com

Verify Firewall/Network Access:

telnet endpoint.example.com 443
nc -zv endpoint.example.com 443

Performance Troubleshooting

Slow Data Delivery

1. Check Buffer Configuration

Adjust buffer settings for better throughput:

nodes:
  - name: my_otlp
    type: otlp_output
    endpoint: endpoint.example.com
    port: 443
    buffer_max_bytesize: 10485760  # 10MB
    buffer_ttl: 15m
    buffer_path: /var/lib/edgedelta/buffers
    parallel_worker_count: 5

2. Increase Parallel Workers

For high-volume workloads:

parallel_worker_count: 10  # Default is 5

3. Monitor Backpressure

Check Edge Delta metrics for:

Buffer usage percentage
Failed delivery attempts
Retry counts

Data Not Arriving

1. Verify Workflow Configuration

Ensure the OTLP destination is properly connected in your workflow:

workflows:
  - name: otlp_workflow
    source_labels:
      - your_source
    destination_labels:
      - your_otlp_destination

2. Check Filters

Verify that filters aren’t blocking data:

workflows:
  - name: otlp_workflow
    source_labels:
      - your_source
    processor_labels:
      - your_processor  # Ensure processor passes data
    destination_labels:
      - your_otlp_destination

3. Enable Debug Logging

Increase log verbosity to see detailed information:

config_version: v2
agent_settings:
  log_level: debug

Best Practices

Configuration Recommendations

Always specify the protocol explicitly
```
protocol: http  # or 'grpc'
```
Use protobuf encoding for HTTP when possible
```
encoding: proto
```
Enable TLS for production endpoints
```
tls:
  enabled: true
```

Configure appropriate buffer settings

buffer_max_bytesize: 10485760
buffer_ttl: 15m
buffer_path: /var/lib/edgedelta/buffers

Set appropriate timeouts
```
timeout: 30s
```

Security Considerations

Store sensitive credentials securely
- Use environment variables or secrets management
- Never commit tokens to version control

Use certificate validation in production

tls:
  enabled: true
  ignore_certificate_check: false
  ca_file: /path/to/ca.pem

Rotate credentials regularly
- Set expiration reminders for API tokens
- Update configurations when credentials are rotated

Getting Help

If issues persist after troubleshooting:

Collect diagnostic information:
- Edge Delta agent version
- Complete OTLP destination configuration (redact sensitive data)
- Error messages from logs
- Network test results
- Destination service type and version
Check Edge Delta documentation:
- OTLP Destination Configuration
- Troubleshooting Cloud Pipelines
Contact Support:
- Edge Delta support portal
- Community forums
- Slack workspace

Provide detailed information about your configuration and the steps you’ve already taken to troubleshoot.

Troubleshooting OTLP Destination

Overview

Quick Diagnostic Checklist

Common Error Messages

Error: “context deadline exceeded”

Symptoms

Causes and Solutions

Error: “401 Unauthorized” or “Failed to authenticate”

Symptoms

Causes and Solutions

Error: “no such host” or URL Parsing Issues

Symptoms

Cause

Solution

Error: “Port appended to URI path”

Symptoms

Cause

Solution

Protocol-Specific Troubleshooting

gRPC Protocol

Common Issues

HTTP Protocol

Common Issues

Destination-Specific Configuration

Dynatrace

Sumo Logic

Grafana Cloud

AWS Distro for OpenTelemetry (ADOT)

Verification and Testing

Check Edge Delta Logs

Test Configuration

Network Diagnostics

Performance Troubleshooting

Slow Data Delivery

Data Not Arriving

Best Practices

Configuration Recommendations

Security Considerations

Getting Help

Edge Delta AI Assistant

Quick Topics

Recent Questions

Hi! I'm your Edge Delta AI Assistant

Current Context