Edge Delta Prometheus Remote Write Source

The Edge Delta Prometheus Remote Write Source node receives metrics pushed to the agent over the Prometheus remote write protocol.

17 minute read

Overview

The Prometheus Remote Write Source node runs an HTTP server that receives metrics pushed to the Edge Delta agent over the Prometheus remote write protocol. Any client that speaks remote write 1.0 can send to it, including Prometheus itself and the OpenTelemetry Collector.

Senders control what is forwarded and when. To have the agent scrape targets instead, use the Prometheus Source.

This node is supported on Edge fleets only. A Cloud Fleet pipeline that uses it fails validation with node type: "prometheus_remote_write_input" is not allowed in: Cloud Fleet.

This node was introduced in v2.22.0.

outgoing_data_types: metric

Securing the Endpoint

The node does not authenticate senders. It has no token or authorization parameter, so anyone who can reach the port can push arbitrary metric names and labels into your pipeline. Incoming labels also overwrite the node’s own configured attributes when the keys collide, so a sender can spoof them.

If the port is reachable beyond a trusted network, restrict access at the network level, or require client certificates:

nodes:
  - name: my_prometheus_remote_write_input
    type: prometheus_remote_write_input
    port: 9201
    tls:
      crt_file: /certs/server-cert.pem
      key_file: /certs/server-key.pem
      ca_file: /certs/ca.pem
      client_auth_type: requireandverifyclientcert

Endpoints

The server exposes the following paths on the configured port:

Path	Method	Purpose
`/api/v1/write`	POST	Receives remote write requests.
`/healthz`	Any	Health check. Returns `200`.
`/_/health`	Any	Health check. Returns `200`.

Any other path returns 404.

Response Codes

Code	Meaning
`204`	The request was accepted.
`400`	The body could not be decoded as a remote write 1.0 request.
`405`	A method other than POST was used on `/api/v1/write`.
`413`	The request body is larger than `drop_limit`.
`415`	The payload is remote write 2.0, or `Content-Encoding` declares an encoding other than snappy.
`429`	The configured `rate_limit` refused the request.
`503`	The node’s internal data channel is at or above 90 percent of `channel_size`.

A sender that receives 429 or 503 should retry. Prometheus retries 5xx responses automatically; whether it retries 429 is controlled by the retry_on_http_429 option in its remote_write configuration.

Protocol Support

The node accepts remote write 1.0 requests: a snappy-compressed, protobuf-encoded prometheus.WriteRequest posted to /api/v1/write. A missing Content-Encoding header is tolerated because some clients omit it, but a header that declares a non-snappy encoding is rejected.

The following are not ingested:

Remote write 2.0 payloads. The node responds with 415, as the remote write 2.0 specification prescribes for receivers that support only 1.0.
Native histograms. A timeseries that carries no float samples is dropped and counted as a node error. Classic histograms sent as _bucket, _sum, and _count series are ingested normally.
Exemplars.

NaN samples are Prometheus staleness markers, so they are skipped without being counted as an error.

Metric Names and Labels

The __name__ label becomes the metric name. Every other label on the timeseries becomes an attribute on the resulting metric item, merged with the node’s own attributes.

Metric Type Mapping

Senders transmit metric metadata in separate remote write requests, so the node caches it by metric family name and applies it to samples that arrive later. The cache holds up to 10,000 families. When metadata is available, the type it reports determines the mapping:

Sender metadata type	Mapped to
`COUNTER`	Monotonic cumulative sum
`GAUGE`	Gauge
`INFO`, `STATESET`	Non-monotonic cumulative sum
Any other type (`HISTOGRAM`, `SUMMARY`, `GAUGEHISTOGRAM`, `UNKNOWN`)	Typed by name suffix, see below

When no metadata has arrived for a metric family, or the metadata reports a type the node does not map directly, the node falls back to the Prometheus naming conventions. Histograms and summaries always take this path, so it is the common case:

Metric name suffix	Mapped to
`_total`, `_count`, `_bucket`, `_sum`	Monotonic cumulative sum
Anything else	Gauge

Metadata also supplies the unit and description on the metric item. If a sender does not transmit metadata, the description is empty and the unit defaults to 1.

Example Configuration

In this example, the node listens on port 9201 across all interfaces and waits up to one minute for a request body to arrive.

nodes:
  - name: my_prometheus_remote_write_input
    type: prometheus_remote_write_input
    listen: 0.0.0.0
    port: 9201
    read_timeout: 1m
    parallel_worker_count: 2
    channel_size: 1000

Configuring Prometheus to Send Data

Add a remote_write block to prometheus.yml that points at the node:

remote_write:
  - url: http://<edge-delta-host>:9201/api/v1/write

Replace <edge-delta-host> with the address of the agent, and the port with the one configured on the node.

Metric metadata supplies the metric type, unit, and description on the Edge Delta side. If metadata sending has been turned off in your configuration, re-enable it so metric types map from the sender’s type rather than from the name suffix:

remote_write:
  - url: http://<edge-delta-host>:9201/api/v1/write
    metadata_config:
      send: true
      send_interval: 1m

Reload or restart Prometheus to apply the change.

Metadata Timing

Prometheus sends metadata on its own schedule, controlled by send_interval, which defaults to 1m. Until the first batch of metadata arrives, the node has nothing cached, so metric types are inferred from name suffixes and descriptions are empty. This window opens when the agent starts and reopens after every agent restart, because the cache is held in memory.

Items already emitted are not corrected once metadata arrives. Lowering send_interval shortens the window.

Required Parameters

name

A descriptive name for the node. This is the name that will appear in pipeline builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: prometheus_remote_write_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

port

The port parameter specifies the port number to listen on. It is specified as an integer and is required. The port must be between 1 and 65535, and must be unique within the pipeline.

A port below 1025 produces a validation warning that data will not flow unless Edge Delta was installed with root privileges. Avoid port 9090 when Prometheus runs on the same host, since that is Prometheus’s own default port.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>

Optional Parameters

disabled

The disabled parameter disables the node in the pipeline. When set to true, the agent ignores the node at startup. You can toggle nodes on or off in the pipeline builder. It is specified as a boolean, defaults to false, and is optional.

nodes:
  - name: <node name>
    type: <node type>
    disabled: true

listen

The listen parameter specifies the address to listen on for incoming traffic. It is specified as a string and is optional. The default is 0.0.0.0.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    listen: <host>

read_timeout

The read_timeout parameter defines how long the server waits to read incoming request data before timing out. It is specified as a duration and is optional. The default is 1m.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    read_timeout: <duration>

write_timeout

The write_timeout parameter sets how long the server waits to finish writing a response before timing out. It is specified as a duration and is optional. The default is 10s.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    write_timeout: <duration>

idle_timeout

The idle_timeout parameter sets how long the server keeps an idle connection open before closing it. It is specified as a duration and is optional. The default is 1m.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    idle_timeout: <duration>

rate_limit

The rate_limit parameter enables you to control data ingestion based on system resource usage. This advanced setting helps prevent source nodes from overwhelming the agent by automatically throttling or stopping data collection when CPU or memory thresholds are exceeded.

Use rate limiting to prevent runaway log collection from overwhelming the agent in high-volume sources, protect agent stability in resource-constrained environments with limited CPU/memory, automatically throttle during bursty traffic patterns, and ensure fair resource allocation across source nodes in multi-tenant deployments.

When rate limiting triggers, pull-based sources (File, S3, HTTP Pull) stop fetching new data, push-based sources (HTTP, TCP, UDP, OTLP) reject incoming data, and stream-based sources (Kafka, Pub/Sub) pause consumption. Rate limiting operates at the source node level, where each source with rate limiting enabled independently monitors and enforces its own thresholds.

Rate limiting vs backpressure: Rate limiting and backpressure are different mechanisms. Rate limiting controls data ingestion at the source based on CPU or memory thresholds. Backpressure controls data output at the destination when in-memory queues fill up. Backpressure logs such as backpressure: dataCh at 92% capacity do not appear when rate limiting triggers, and rate limiting logs do not appear during backpressure events.

Observability: When rate limiting is active in enforce mode, the agent does not produce INFO, WARN, or ERROR level logs. To confirm that rate limiting is triggering, check the source node’s Bytes I/O metrics in the Edge Delta app for gaps or drops that coincide with expected throttling periods.

Configuration Steps:

Click Add New in the Rate Limit section
Click Add New for Evaluation Policy
Select Policy Type:

CPU Usage: Monitors CPU consumption and rate limits when usage exceeds defined thresholds. Use for CPU-intensive sources like file parsing or complex transformations.
Memory Usage: Monitors memory consumption and rate limits when usage exceeds defined thresholds. Use for memory-intensive sources like large message buffers or caching.
AND (composite): Combines multiple sub-policies with AND logic. All sub-policies must be true simultaneously to trigger rate limiting. Use when you want conservative rate limiting (both CPU and memory must be high).
OR (composite): Combines multiple sub-policies with OR logic. Any sub-policy can trigger rate limiting. Use when you want aggressive rate limiting (either CPU or memory being high triggers).

Select Evaluation Mode. Choose how the policy behaves when thresholds are exceeded:

Enforce (default): Actively applies rate limiting when thresholds are met. Pull-based sources (File, S3, HTTP Pull) stop fetching new data, push-based sources (HTTP, TCP, UDP, OTLP) reject incoming data, and stream-based sources (Kafka, Pub/Sub) pause consumption. Use in production to protect agent resources.
Monitor: Logs when rate limiting would occur without actually limiting data flow. Use for testing thresholds before enforcing them in production.
Passthrough: Disables rate limiting entirely while keeping the configuration in place. Use to temporarily disable rate limiting without removing configuration.

Set Absolute Limits and Relative Limits (for CPU Usage and Memory Usage policies)

Note: If you specify both absolute and relative limits, the system evaluates both conditions and rate limiting triggers when either condition is met (OR logic). For example, if you set absolute limit to 1.0 CPU cores and relative limit to 50%, rate limiting triggers when the source uses either 1 full core OR 50% of available CPU, whichever happens first.

For CPU Absolute Limits: Enter value in full core units:
- 0.1 = one-tenth of a CPU core
- 0.5 = half a CPU core
- 1.0 = one full CPU core
- 2.0 = two full CPU cores
For CPU Relative Limits: Enter percentage of total available CPU (0-100):
- 50 = 50% of available CPU
- 75 = 75% of available CPU
- 85 = 85% of available CPU
For Memory Absolute Limits: Enter value in bytes
- 104857600 = 100Mi (100 × 1024 × 1024)
- 536870912 = 512Mi (512 × 1024 × 1024)
- 1073741824 = 1Gi (1 × 1024 × 1024 × 1024)
For Memory Relative Limits: Enter percentage of total available memory (0-100)
- 60 = 60% of available memory
- 75 = 75% of available memory
- 80 = 80% of available memory

Set Refresh Interval (for CPU Usage and Memory Usage policies). Specify how frequently the system checks resource usage:

Recommended Values:
- 10s to 30s for most use cases
- 5s to 10s for high-volume sources requiring quick response
- 1m or higher for stable, low-volume sources

The system fetches current CPU/memory usage at the specified refresh interval and uses that value for evaluation until the next refresh. Shorter intervals provide more responsive rate limiting but incur slightly higher overhead, while longer intervals are more efficient but slower to react to sudden resource spikes.

The GUI generates YAML as follows:

# Simple CPU-based rate limiting
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: cpu_usage
        evaluation_mode: enforce
        absolute_limit: 0.5  # Limit to half a CPU core
        refresh_interval: 10s

# Simple memory-based rate limiting
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: memory_usage
        evaluation_mode: enforce
        absolute_limit: 536870912  # 512Mi in bytes
        refresh_interval: 30s

Composite Policies (AND / OR)

When using AND or OR policy types, you define sub-policies instead of limits. Sub-policies must be siblings (at the same level)—do not nest sub-policies within other sub-policies. Each sub-policy is independently evaluated, and the parent policy’s evaluation mode applies to the composite result.

AND Logic: All sub-policies must evaluate to true at the same time to trigger rate limiting. Use when you want conservative rate limiting (limit only when CPU AND memory are both high).
OR Logic: Any sub-policy evaluating to true triggers rate limiting. Use when you want aggressive protection (limit when either CPU OR memory is high).

Configuration Steps:

Select AND (composite) or OR (composite) as the Policy Type
Choose the Evaluation Mode (typically Enforce)
Click Add New under Sub-Policies to add the first condition
Configure the first sub-policy by selecting policy type (CPU Usage or Memory Usage), selecting evaluation mode, setting absolute and/or relative limits, and setting refresh interval
In the parent policy (not within the child), click Add New again to add a sibling sub-policy
Configure additional sub-policies following the same pattern

The GUI generates YAML as follows:

# AND composite policy - both CPU AND memory must exceed limits
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: and
        evaluation_mode: enforce
        sub_policies:
          # First sub-policy (sibling)
          - policy_type: cpu_usage
            evaluation_mode: enforce
            absolute_limit: 0.75  # Limit to 75% of one core
            refresh_interval: 15s
          # Second sub-policy (sibling)
          - policy_type: memory_usage
            evaluation_mode: enforce
            absolute_limit: 1073741824  # 1Gi in bytes
            refresh_interval: 15s

# OR composite policy - either CPU OR memory can trigger
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: or
        evaluation_mode: enforce
        sub_policies:
          - policy_type: cpu_usage
            evaluation_mode: enforce
            relative_limit: 85  # 85% of available CPU
            refresh_interval: 20s
          - policy_type: memory_usage
            evaluation_mode: enforce
            relative_limit: 80  # 80% of available memory
            refresh_interval: 20s

# Monitor mode for testing thresholds
nodes:
  - name: <node name>
    type: <node type>
    rate_limit:
      evaluation_policy:
        policy_type: memory_usage
        evaluation_mode: monitor  # Only logs, doesn't limit
        relative_limit: 70  # Test at 70% before enforcing
        refresh_interval: 30s

channel_size

YAML Only

The channel_size parameter sets the size of the internal data channel buffer. It is specified as an integer, has a default of 1000, and is optional.

parallel_worker_count

YAML Only

The parallel_worker_count parameter sets the number of workers that process incoming data in parallel. It is specified as an integer, has a default of 2, and is optional.

source_metadata

This option is used to define which detected resources and attributes to add to each data item as it is ingested by Edge Delta. You can select:

Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
Default: This option includes the required resources and attributes plus those selected by Edge Delta
High: This option includes the required resources and attributes along with a larger selection of common optional fields.
Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.

max_connections

The max_connections parameter sets the maximum number of concurrent connections the server accepts. It is specified as an integer and is optional. The default is 262144.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    max_connections: <count>

max_connections_per_ip

The max_connections_per_ip parameter sets the maximum concurrent connections from a single IP address. Set to 0 for unlimited. It is specified as an integer and is optional. The default is 0 (unlimited).

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    max_connections_per_ip: <count>

read_buffer_size

The read_buffer_size parameter sets the per-connection buffer size for reading request data. This also controls the maximum header size. It is specified as a size and is optional. The default is 16KB.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    read_buffer_size: <size>

write_buffer_size

The write_buffer_size parameter sets the per-connection buffer size for writing response data. It is specified as a size and is optional. The default is 4KB.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    write_buffer_size: <size>

drop_limit

The drop_limit parameter sets the maximum request body size. The server rejects a larger request with 413 before reading the body, and records the rejection in the ed.pipeline.node.dropped_items.body_too_large metric. It is specified as a size and is optional. The default is 128MB.

Regardless of this setting, the node rejects a request whose snappy length prefix claims more than 256 MB decompressed, or more than the snappy format could produce from the bytes received.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    drop_limit: <size>

tls

The tls parameter is a dictionary type that enables a number of TLS options to be set using sub-parameters.

Providing a tls block switches the server from HTTP to HTTPS. There is no separate protocol setting on this node.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      crt_file: /certs/server-cert.pem
      key_file: /certs/server-key.pem

ca_file The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      ca_file: /certs/ca.pem

ca_path The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      ca_path: <path>

client_auth_type The client_auth_type parameter is a child of the tls parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.

The following authentication methods are available:

noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      client_auth_type: <auth type>

crt_file The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      crt_file: /certs/server-cert.pem

key_file The key_file parameter is a child of the tls parameter. It specifies the private key file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      key_file: /certs/server-key.pem

key_password The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      key_password: <password>

max_version The max_version parameter is a child of the tls parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.

You can select one of the following options:

TLSv1_0
TLSv1_1
TLSv1_2
TLSv1_3

nodes:
  - name: <node name>
    type: prometheus_remote_write_input
    port: <port number>
    tls:
      max_version: <TLS version>

min_version The min_version parameter is a child of the tls parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2.