Edge Delta Gateway Connection

Overview

The Gateway Connection Node is designed to send various data types, such as logs, metrics, and traces, to an Edge Delta Gateway fleet. This node supports advanced TLS configurations for secure data transmission.

• incoming_data_types: log, metric, trace

See:

• Kubernetes Deployment Examples
• Integrate Edge Delta’s Edge, Coordinator, and Gateway Pipelines
• Edge Delta Pipeline Source

Example Configuration

In this example, the gateway connection node is configured to connect to an in-cluster gateway using its name. This automatically sets the node to use the k8s_service_name of the gateway to handle communication on port 442 using the gRPC protocol.

Note: The Gateway option is in the user interface only. It is used to automatically populate the other fields, which appear in the YAML. With this field, you can select the gateway pipeline and node by name, or you can select Custom and specify the integration manually.

nodes:
- name: ed_gateway_output_8e59
  type: ed_gateway_output
  metadata: '{"pipelineId":"a503a449-f54e-4c86-9d55-6a6353cc3f2d","nodeId":"ed_gateway_input_006e"}'
  user_description: ED Gateway Connection
  port: 443
  protocol: grpc
  endpoint_resolution_type: k8s
  k8s_service_name: pd3b496512024855-gw-svc

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: ed_gateway_output

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

port

The port parameter specifies the port number to send outgoing data, ranging from 1 to 65535. When a named gateway is selected this defaults to 443. It is only editable when the gateway is set to Custom.

nodes:
- name: ed_gateway_output_8e59
  type: ed_gateway_output
  port: 443

protocol

The protocol parameter specifies the protocol used to send data. It is specified as a string. You must set grpc or http.

nodes:
- name: ed_gateway_output_8e59
  type: ed_gateway_output
  protocol: grpc

endpoint_resolution_type

This parameter specifies how to discover endpoints for the load balancer. Options include k8s, dns, or static. When a named gateway is selected this defaults to k8s.

nodes:
- name: ed_gateway_output_8e59
  type: ed_gateway_output
  endpoint_resolution_type: k8s

hostname

The hostname parameter is applicable only when the Gateway is set to Custom and the endpoint_resolution_type is set to DNS. It specifies the domain name or address of the service or endpoint to which you are sending data. This domain name is what will be translated into an IP address or addresses by a DNS server.

nodes:
  - name: gateway_connection
    type: ed_gateway_output
    endpoint_resolution_type: dns
    hostname: example.hostname.com

endpoints

The endpoints parameter specifies a list of static IP addresses or URLs to which data should be sent. This parameter is available only when the gateway is set to “Custom” and the endpoint_resolution_type is set to “Static”. It allows for direct specification of destination addresses without requiring DNS resolution.

nodes:
  - name: gateway_connection
    type: ed_gateway_output
    gateway_type: custom
    endpoint_resolution_type: static
    endpoints:
      - 192.168.1.10
      - 192.168.1.11
      - https://example.endpoint.com

This parameter is beneficial when you want to ensure that data is routed to fixed addresses, providing a stable route for data transmission without relying on dynamic DNS.

pipeline

The pipeline parameter is visible only when the Gateway type is set to Custom. It specifies the particular gateway to associate the destination node with in the user interface:

nodes:
  - name: gateway_connection
    type: ed_gateway_output
    gateway_type: custom
    pipeline: example-pipeline

interval

The interval parameter is used to define the time duration between subsequent DNS queries for resolving the IP address of the hostname. This parameter is visible only when the Gateway is set to Custom and the endpoint_resolution_type is set to DNS. It allows you to specify how frequently the DNS resolution should occur, which can help in dynamically handling changes in IP addresses for the gateway you’re connecting to.

nodes:
  - name: gateway_connection
    type: ed_gateway_output
    gateway_type: custom
    endpoint_resolution_type: dns
    interval: 5m

Optional Parameters

target_allocation_type

Specifies how to send data from the load balancer. Options include consistent or round_robin. The default is consistent.

• Consistent: The edge pipeline sends its data to the same gateway instance, determined by a consistent hashing algorithm. This keeps related data together, which improves the accuracy of deduplication and aggregation. It also reduces reprocessing overhead for the same data source. Use Consistent when processing depends on maintaining data locality—such as metric aggregation and deduplication.
• Round Robin: The edge pipeline distributes its data evenly across all available gateway instances, one after the other. This balances the load evenly across gateways, regardless of data source. Use this option when the gateway pipeline has been scaled to multiple instances. Use this option when you have specified more than one instance in a custom gateway configuration, where Endpoint Resolution Type is set to Static and you specified multiple Endpoints, or where Endpoint Resolution Type is set to DNS and you specified multiple Hostnames.

nodes:
- name: gateway_connection
  type: ed_gateway_output
  target_allocation_type: consistent

buffer_ttl

The buffer_ttl parameter defines how long output data should be retried if streaming fails. It is specified as a duration.

nodes:
- name: gateway_connection
  type: ed_gateway_output
  buffer_ttl: 15m

This setting defines how long the agent will keep unsent data in the buffer if streaming to the destination fails. If the connection is interrupted, the agent stores the data locally. The agent will continue retrying to send the data for up to the specified TTL duration. After this time expires, the data is discarded. Use this to prevent data loss during temporary network issues, while avoiding unlimited retry loops that consume disk space.

buffer_path

Specifies the path where output is saved if streaming fails, to be retried later.

nodes:
- name: gateway_connection
  type: ed_gateway_output
  buffer_path: /var/log/edgedelta/outputbuffer

This is the file system path where the agent writes unsent output data when streaming fails. The buffer is stored here until it can be retried. This path must be writable and should have enough disk space to handle potential buffering during outages. Use this option to control where buffered data is saved so it can survive agent restarts or be monitored for troubleshooting.

buffer_max_bytesize

Defines the maximum size of the buffer used to store data if streaming fails.

nodes:
- name: gateway_connection
  type: ed_gateway_output
  buffer_max_bytesize: 10MB

This setting limits how much data can be buffered before the agent stops accepting new data into the buffer. Once this size is reached, older data may be dropped (depending on internal behavior) or buffering may pause until space frees up. Use this option to prevent excessive disk usage and ensure the buffer doesn’t grow unbounded during prolonged connection failures.

tls

The tls configuration offers options for setting up secure SSL/TLS connections.

nodes:
- name: gateway_connection
  type: ed_gateway_output
  tls:
    enabled: true
    ca_file: /path/to/ca.crt
    crt_file: /path/to/client.crt
    key_file: /path/to/client.key
    ignore_certificate_check: false

enabled Enables SSL/TLS for secure connections. Defaults to false.

ignore_certificate_check Disables certificate verification. Use with caution; defaults to false.

ca_file Specifies the file path to the CA certificate for verifying server connections.

crt_file Specifies the path to the client certificate for authentication.

key_file Specifies the path to the client’s private key for authentication.

For more detailed TLS options, you can configure client_auth_type, min_version, and max_version.