Edge Delta Prometheus Remote Write Destination

Overview

This node requires Edge Delta agent version v1.26.0 or higher.

The Edge Delta Prometheus remote write destination node sends collected metrics directly to a specified Prometheus endpoint using the remote write protocol. This allows for centralized metrics storage and analysis without the need for Prometheus to scrape metrics from each monitored service individually.

• incoming_data_types: metric, heartbeat.

The prometheus_remote_write_output is useful in situations where network security policies prevent services from being scraped directly or in architectures where you wish to have a centralized Prometheus setup receiving metrics from various remote nodes or regions. This is especially beneficial when dealing with large distributed systems where direct scraping might not be efficient or feasible.

In contrast, the prometheus_exporter_output is suitable for scenarios where you want to expose your metrics to a Prometheus server for scraping. This is often used if you are deploying services that Prometheus will monitor directly, and you aim to use the metric data for alerting or visualization.

Example Configuration

This example configuration shows how to set up the Prometheus remote write destination node in an Edge Delta pipeline.

nodes:
- name: prometheus_remote_write_output
  type: prometheus_remote_write_output
  endpoint: https://localhost:8087/api/v1/write

Required Parameters

name

A descriptive name for the node. This is the name that will appear in pipeline builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: prometheus_exporter_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

endpoint

The endpoint parameter defines the URL where Edge Delta will send metrics data. The endpoint format should adhere to the Prometheus remote write API.

nodes:
- name: prometheus_remote_write_output
  type: prometheus_remote_write_output
  endpoint: https://localhost:8087/api/v1/write

Optional Parameters

headers

Minimum Agent Version: v2.8.0

The headers parameter allows you to add custom HTTP headers to all requests sent to the Prometheus remote write endpoint. This is useful for passing static authentication tokens, API keys, or other custom metadata required by your Prometheus-compatible endpoint. It is specified as an array of header/value pairs and is optional.

nodes:
- name: prometheus_remote_write_output
  type: prometheus_remote_write_output
  endpoint: https://prometheus.example.com/api/v1/write
  headers:
    - header: X-API-Key
      value: my-api-key
    - header: X-Custom-Header
      value: custom-value

authorization

Minimum Agent Version: v2.8.0

The authorization parameter enables OAuth2.0 Client Credentials Flow authentication for secure metric forwarding to authenticated Prometheus-compatible endpoints. When configured, Edge Delta automatically obtains and refreshes OAuth tokens for requests to the remote write endpoint. It is specified as an object and is optional.

Child Parameters:

client_credentials Child Parameters:

nodes:
- name: prometheus_remote_write_output
  type: prometheus_remote_write_output
  endpoint: https://prometheus.example.com/api/v1/write
  authorization:
    strategy: oauth2.0_client_credentials
    client_credentials:
      client_id: my-client-id
      client_secret: ${OAUTH_CLIENT_SECRET}
      token_url: https://auth.example.com/oauth/token
      scopes:
        - metrics:write

Example with extra parameters:

nodes:
- name: prometheus_remote_write_output
  type: prometheus_remote_write_output
  endpoint: https://prometheus.example.com/api/v1/write
  authorization:
    strategy: oauth2.0_client_credentials
    client_credentials:
      client_id: my-client-id
      client_secret: ${OAUTH_CLIENT_SECRET}
      token_url: https://auth.example.com/oauth/token
      scopes:
        - metrics:write
        - metrics:read
      extra_parameters:
        - key: audience
          value: https://prometheus.example.com

tls

Configure TLS settings for secure connections to this destination. TLS is optional and typically used when connecting to endpoints that require encrypted transport (HTTPS) or mutual TLS.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      <tls options>

Enable TLS

Enables TLS encryption for outbound connections to the destination endpoint. When enabled, all communication with the destination will be encrypted using TLS/SSL. This should be enabled when connecting to HTTPS endpoints or any service that requires encrypted transport. (YAML parameter: enabled)

Default: false

When to use: Enable when the destination requires HTTPS or secure connections. Always enable for production systems handling sensitive data, connections over untrusted networks, or when compliance requirements mandate encryption in transit.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      enabled: true

Ignore Certificate Check

Disables TLS certificate verification, allowing connections to servers with self-signed, expired, or invalid certificates. This bypasses security checks that verify the server’s identity and certificate validity. (YAML parameter: ignore_certificate_check)

Default: false

When to use: Only use in development or testing environments with self-signed certificates. NEVER enable in production—this makes your connection vulnerable to man-in-the-middle attacks. For production with self-signed certificates, use ca_file or ca_path to explicitly trust specific certificates instead.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      ignore_certificate_check: true  # Only for testing!

CA Certificate File

Specifies the absolute path to a CA (Certificate Authority) certificate file used to verify the destination server’s certificate. This allows you to trust specific CAs beyond the system’s default trusted CAs, which is essential when connecting to servers using self-signed certificates or private CAs. (YAML parameter: ca_file)

When to use: Required when connecting to servers with certificates signed by a private/internal CA, or when you want to restrict trust to specific CAs only. Choose either ca_file (single CA certificate) or ca_path (directory of CA certificates), not both.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      ca_file: /certs/ca.pem

CA Certificate Path

Specifies a directory path containing one or more CA certificate files for verifying the destination server’s certificate. Use this when you need to trust multiple CAs or when managing CA certificates across multiple files. All certificate files in the directory will be loaded. (YAML parameter: ca_path)

When to use: Alternative to ca_file when you have multiple CA certificates to trust. Useful for environments with multiple private CAs or when you need to rotate CA certificates without modifying configuration. Choose either ca_file or ca_path, not both.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      ca_path: /certs/ca-certificates/

Certificate File

Path to the client certificate file (public key) used for mutual TLS (mTLS) authentication with the destination server. This certificate identifies the client to the server and must match the private key. The certificate should be in PEM format. (YAML parameter: crt_file)

When to use: Required only when the destination server requires mutual TLS authentication, where both client and server present certificates. Must be used together with key_file. Not needed for standard client TLS connections where only the server presents a certificate.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      crt_file: /certs/client-cert.pem
      key_file: /certs/client-key.pem

Private Key File

Path to the private key file corresponding to the client certificate. This key must match the public key in the certificate file and is used during the TLS handshake to prove ownership of the certificate. Keep this file secure with restricted permissions. (YAML parameter: key_file)

When to use: Required for mutual TLS authentication. Must be used together with crt_file. If the key file is encrypted with a password, also specify key_password. Only needed when the destination server requires client certificate authentication.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      crt_file: /certs/client-cert.pem
      key_file: /certs/client-key.pem
      key_password: <password>  # Only if key is encrypted

Private Key Password

Password (passphrase) used to decrypt an encrypted private key file. Only needed if your private key file is password-protected. If your key file is unencrypted, omit this parameter. (YAML parameter: key_password)

When to use: Optional. Only required if key_file is encrypted/password-protected. For enhanced security, use encrypted keys in production environments. If you receive “bad decrypt” or “incorrect password” errors, verify the password matches the key file encryption.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      crt_file: /certs/client-cert.pem
      key_file: /certs/encrypted-client-key.pem
      key_password: mySecurePassword123

Minimum TLS Version

Minimum TLS protocol version to use when connecting to the destination server. This enforces a baseline security level by refusing to connect if the server doesn’t support this version or higher. (YAML parameter: min_version)

Available versions:

• TLSv1_0 - Deprecated, not recommended (security vulnerabilities)
• TLSv1_1 - Deprecated, not recommended (security vulnerabilities)
• TLSv1_2 - Recommended minimum for production (default)
• TLSv1_3 - Most secure, use when destination supports it

Default: TLSv1_2

When to use: Set to TLSv1_2 or higher for production deployments. Only use TLSv1_0 or TLSv1_1 if connecting to legacy servers that don’t support newer versions, and be aware of the security risks. TLS 1.0 and 1.1 are officially deprecated.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      min_version: TLSv1_2

Maximum TLS Version

Maximum TLS protocol version to use when connecting to the destination server. This is typically used to restrict newer TLS versions if compatibility issues arise with specific server implementations. (YAML parameter: max_version)

Available versions:

• TLSv1_0
• TLSv1_1
• TLSv1_2
• TLSv1_3

When to use: Usually left unset to allow the most secure version available. Only set this if you encounter specific compatibility issues with TLS 1.3 on the destination server, or for testing purposes. In most cases, you should allow the latest TLS version.

YAML Configuration Example:

nodes:
  - name: <node name>
    type: <destination type>
    tls:
      max_version: TLSv1_3

persistent_queue

The persistent_queue configuration enables disk-based buffering to prevent data loss during destination failures or slowdowns. When enabled, the agent stores data on disk and automatically retries delivery when the destination recovers.

Complete example:

persistent_queue:
  path: /var/lib/edgedelta/outputbuffer
  mode: error
  max_byte_size: 1GB
  drain_rate_limit: 1000

How it works:

1. Normal operation: Data flows directly to the destination (for error and backpressure modes) or through the disk buffer (for always mode)
2. Destination issue detected: Based on the configured mode, data is written to disk at the configured path
3. Recovery: When the destination recovers, buffered data drains at the configured drain_rate_limit while new data continues flowing
4. Completion: Buffer clears and normal operation resumes

Key benefits:

• Data durability: Logs preserved during destination outages and slowdowns
• Agent protection: Slow backends don’t cascade failures into the agent cluster
• Automatic recovery: No manual intervention required
• Configurable behavior: Choose when and how buffering occurs based on your needs

path

The path parameter specifies the directory where buffered data is stored on disk. This parameter is required when configuring a persistent queue.

Example:

persistent_queue:
  path: /var/lib/edgedelta/outputbuffer

Requirements:

• Required field - persistent queue will not function without a valid path
• The directory must have sufficient disk space for the configured max_byte_size
• The agent process must have read/write permissions to this location
• The path should be on a persistent volume (not tmpfs or memory-backed filesystem)

Best practices:

• Use dedicated storage for buffer data separate from logs
• Monitor disk usage to prevent buffer from filling available space
• Ensure the path persists across agent restarts to maintain buffered data

max_byte_size

The max_byte_size parameter defines the maximum disk space the persistent buffer is allowed to use. Once this limit is reached, any new incoming items are dropped, ensuring the buffer never grows beyond the configured maximum.

Example:

persistent_queue:
  path: /var/lib/edgedelta/outputbuffer
  max_byte_size: 1GB

Sizing guidance:

• Small deployments (1-10 logs/sec): 100MB - 500MB
• Medium deployments (10-100 logs/sec): 500MB - 2GB
• Large deployments (100+ logs/sec): 2GB - 10GB

Calculation example:

Average log size: 1KB
Expected outage duration: 1 hour
Log rate: 100 logs/sec

Buffer size = 1KB × 100 logs/sec × 3600 sec = 360MB
Recommended: 500MB - 1GB (with safety margin)

Important: Set this value based on your disk space availability and expected outage duration. The buffer will accumulate data during destination failures and drain when the destination recovers.

mode

The mode parameter determines when data is buffered to disk. Three modes are available:

• error (default) - Buffers data only when the destination returns errors (connection failures, HTTP 5xx errors, timeouts). During healthy operation, data flows directly to the destination without buffering.

• backpressure - Buffers data when the in-memory queue reaches 80% capacity OR when destination errors occur. This mode helps handle slow destinations that respond successfully but take longer than usual to process requests.

• always - Uses write-ahead-log behavior where all data is written to disk before being sent to the destination. This provides maximum durability but adds disk I/O overhead to every operation.

Example:

persistent_queue:
  path: /var/lib/edgedelta/outputbuffer
  mode: error
  max_byte_size: 1GB

Mode comparison:

Why choose error mode:

The error mode provides the minimal protection layer needed to prevent data loss when destinations temporarily fail. Without any persistent queue, a destination outage means data is lost. With error mode enabled, data is preserved on disk during failures and delivered automatically when the destination recovers.

Why choose backpressure mode:

The backpressure mode provides everything error mode offers, plus protection against slow destinations. When a destination is slow but not completely down:

• Without backpressure: Data delivery becomes unreliable, and the backend’s slowness propagates to the agent—the agent can get stuck waiting before sending subsequent payloads
• With backpressure: The agent spills data to disk and continues processing, isolating itself from the slow backend

This prevents a slow destination from cascading failures into your agent cluster. For most production environments, backpressure provides the best balance of protection and performance.

Why choose always mode:

The always mode is designed for customers with extremely strict durability requirements. It forces the agent to write every item to disk before attempting delivery, then reads from disk for transmission. This guarantees that data survives even sudden agent crashes or restarts.

Important: This mode introduces a measurable performance cost. Each agent performs additional disk I/O on every item, which reduces overall throughput. Most deployments do not require this level of durability—this feature addresses specialized needs that apply to a small minority of customers.

Only enable always mode if you have a specific, well-understood requirement where the durability guarantee outweighs the throughput reduction.

strict_ordering

The strict_ordering parameter controls how items are consumed from the persistent buffer.

When strict_ordering: true, the agent runs in strict ordering mode with a single processing thread. This mode always prioritizes draining buffered items first—new incoming data waits until all buffered items are processed in exact chronological order. When strict_ordering: false (default), multiple workers process data in parallel, and new data flows directly to the destination while buffered data drains in the background.

Example:

persistent_queue:
  path: /var/lib/edgedelta/outputbuffer
  mode: always
  strict_ordering: true
parallel_workers: 1

Default value: false

Important: Strict ordering is a specialized feature needed by a very small minority of deployments. Most users should keep the default value of false. Only enable strict ordering if you have a specific, well-understood requirement for exact event sequencing.

Required setting: When strict_ordering: true, you must set parallel_workers: 1. Pipeline validation will fail if parallel_workers is greater than 1 because parallel processing inherently breaks ordering guarantees.

Behavior:

Why the default is false:

In most observability use cases, data freshness is more valuable than strict ordering. When a destination recovers from an outage, operators typically want to see current system state on dashboards immediately, while historical data backfills in the background. The default behavior prioritizes this real-time visibility.

When to enable strict ordering:

Strict ordering is primarily needed by security-focused customers who build systems where events must arrive in the exact delivery order. These customers typically run stateful security streaming engines that depend on precise temporal sequencing.

Specific use cases:

• Stateful security streaming engines - Security systems that maintain state across events and detect patterns based on exact event order
• Audit and compliance logs - Regulatory requirements that mandate audit trails preserve exact temporal sequence
• State reconstruction - Systems that replay events to rebuild state require chronological order

When to keep default (false):

The vast majority of deployments should keep the default:

• Real-time monitoring dashboards - Current state visibility is more important than historical order
• High-volume log ingestion - Faster drain times reduce recovery period
• Stateless analytics - When each log is analyzed independently without temporal correlation

drain_rate_limit

The drain_rate_limit parameter controls the maximum items per second when draining the persistent buffer after a destination recovers from a failure.

Example:

persistent_queue:
  path: /var/lib/edgedelta/outputbuffer
  drain_rate_limit: 1000

Default value: 0 (no limit - drain as fast as the destination accepts)

Why rate limiting matters:

When a destination recovers from an outage, it may still be fragile. Immediately flooding it with hours of backlogged data can trigger another failure. The drain rate limit allows gradual, controlled recovery that protects destination stability.

Choosing the right rate:

Impact on recovery time:

Buffer size: 1GB
Average log size: 1KB
Total items: ~1,000,000 logs

At unlimited (0): Depends on destination capacity
At 5000:      ~3.5 minutes to drain
At 1000:      ~17 minutes to drain
At 100:       ~2.8 hours to drain