Edge Delta Pipeline Source

Overview

The ED Pipeline Source node facilitates the configuration of a gateway pipeline that is designed to consume and process traffic from one or more node pipelines. These pipelines can be situated either within the same cluster or in a different one. This node is crucial for enabling the centralization and processing of data flows from distributed edge sources through the Edge Delta platform.

• incoming_data_types: log, metric, trace

See:

• Kubernetes Deployment Examples
• Integrate Edge Delta’s node, Coordinator, and Gateway Pipelines
• Edge Delta Gateway Connection

This node requires Edge Delta agent version v2.0.0 or higher.

Configuration

The basic configuration for an ED Pipeline Source node includes the primary parameters needed to set up the listening endpoint. It includes a port parameter, where users can specify the port number, such as 443, to which the node will listen for incoming data. The protocol parameter is required and should be set to grpc, which is the standard protocol used for communication with the pipeline source. Additionally, a read_timeout parameter can be set, defaulting to 10s, to determine how long the node should wait for incoming data before timing out.

nodes:
- name: ed_gateway_input_006e
  type: ed_gateway_input
  user_description: ED Pipeline Source
  port: 443
  protocol: grpc
  read_timeout: 1m0s

Required Parameters

name

A descriptive name for the node. This is the name that will appear in pipeline builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ed_gateway_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Port

The port parameter specifies the port number to listen on. It is specified as an integer and is required.

nodes:
  - name: <node name>
    type: ed_gateway_input
    port: <port number>

protocol

The protocol parameter specifies the method of communication to the Pipeline Source. The default value is grpc, and it is required.

nodes:
  - name: <node name>
    type: ed_gateway_input
    port: <port number>
    protocol: <protocol>

Optional Parameters

disabled

The disabled parameter disables the node in the pipeline. When set to true, the agent ignores the node at startup. You can toggle nodes on or off in the pipeline builder. It is specified as a boolean, defaults to false, and is optional.

nodes:
  - name: <node name>
    type: <node type>
    disabled: true

read_timeout

The read_timeout parameter indicates how long to wait for incoming data before timing out. It is specified as a duration and is optional. The default value is 10s.

nodes:
  - name: <node name>
    type: ed_gateway_input
    port: <port number>
    read_timeout: <duration>

channel_size

The channel_size parameter sets the size of the internal data channel buffer. It is specified as an integer, has a default of 1000, and is optional.

parallel_worker_count

The parallel_worker_count parameter sets the number of workers that process incoming data in parallel. It is specified as an integer, has a default of 2, and is optional.

source_metadata

This option is used to define which detected resources and attributes to add to each data item as it is ingested by Edge Delta. You can select:

• Required Only: This option includes the minimum required resources and attributes for Edge Delta to operate.
• Default: This option includes the required resources and attributes plus those selected by Edge Delta
• High: This option includes the required resources and attributes along with a larger selection of common optional fields.
• Custom: With this option selected, you can choose which attributes and resources to include. The required fields are selected by default and can’t be unchecked.

Based on your selection in the GUI, the source_metadata YAML is populated as two dictionaries (resource_attributes and attributes) with Boolean values.

See Choose Data Item Metadata for more information on selecting metadata.

tls

The tls parameter is a dictionary type that enables a number of TLS options to be set using sub-parameters.

nodes:
  - name: <node name>
    type: ed_gateway_input
    port: <port number>
    tls:
      <tls options>:

ca_file The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

ca_path The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

client_auth_type The client_auth_type parameter is a child of the tls parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.

The following authentication methods are available:

• noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
• requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
• requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
• verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
• requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client

crt_file The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

ignore_certificate_check The ignore_certificate_check parameter is a child of the tls parameter. When set to true, it ignores certificate checks for the remote endpoint. It is specified as a Boolean value and the default is false, indicating that TLS verification will be performed. This is an optional parameter.

key_file The key_file parameter is a child of the tls parameter. It specifies the private key file. It is specified as a string and is optional.

key_password The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

max_version The max_version parameter is a child of the tls parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.

You can select one of the following options:

• TLSv1_0
• TLSv1_1
• TLSv1_2
• TLSv1_3

min_version The min_version parameter is a child of the tls parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2.

You can select one of the following options:

• TLSv1_0
• TLSv1_1
• TLSv1_2
• TLSv1_3

Buffer Configuration

The Pipeline source node uses an internal buffer to queue incoming data before processing. If incoming data from node pipelines exceeds the processing capacity, the buffer can fill up, causing data loss. When this happens, the agent logs an error:

log processing buffer is full (capacity: 1000), dropping 1 entries. increase buffer size or reduce send rate

To resolve buffer full errors:

1. Increase parallel workers on the gateway source: Set parallel_worker_count to increase processing throughput. A common starting point is 2× the number of CPU cores allocated to the gateway. The default is 2 workers.

2. Increase buffer size on the gateway source: Set channel_size to increase the buffer capacity. The default is 1000 entries. For high-throughput environments, start with 5000–10000 and increase if buffer full errors persist.

3. Increase parallel workers on the sending side: On the node pipelines, increase parallel_worker_count on the gateway destination node. This spreads the load across more connections but increases resource usage on the sending agents.

Note: The ED_OTLP_TAILER_BUFFER_SIZE environment variable also controls buffer size but is deprecated. Use the node-level channel_size parameter instead. Node-level settings take precedence over environment variables.