Edge Delta Fluentd Source

Ingest logs in fluentd forward protocol.

  6 minute read  

Beta

Overview

The Fluentd node enables the agent to ingest logs in the Fluentd forward protocol.

  • outgoing_data_types: log

Preparing Fluentd

You can use the following configuration in Fluentd to send log data to Edge Delta.

fluentd.conf

<match **>
      https://docs.fluentd.org/v1.0/articles/in_forward
      @type forward
      heartbeat_type none
      keepalive true
      <security>
            shared_key "sharedKey"  <!--shared key is used between nodes.-->
      </security>
      <server>
            host IP or host <!--to an endpoint which distributes the requests to the ED Agents.-->
            port 3421 <!--Port on which the ED Agents listens for Fluentd sources.-->
      </server>
</match>

Example Configuration

nodes:
  - name: my_fluentd_input
    type: fluentd_input
    port: 9898
    read_timeout: 30s
    shared_key: test_shared_key

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: fluentd_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

port

The port parameter specifies the port number to listen on. It is specified as an integer and is required.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>

Optional Parameters

add_ingestion_time

The add_ingestion_time parameter specifies whether to ingest the timestamp. The input must be JSON. It is specified with a Boolean, the default is false and it is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    add_ingestion_time: true

features

YAML Only

The features parameter specifies the data types enabled for this streaming endpoint. You can select the following, some of which are enabled by default:

  • metric (default)
  • edac (default)
  • cluster (default)
  • cluster_pattern
  • cluster_sample
  • log (default)
  • topk (default)
  • alert (default)
  • health
nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    features:
      - cluster_sample

listen

The listen parameter is used to specify the address to listen to for incoming traffic. It is specified as a string, the default value is 0.0.0.0. and it is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    listen: <host>

read_size

YAML Only

The read_size parameter specifies the number of lines to read from the incoming data. If not specified, the default is 1. It is specified as an integer and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    read_size: <int>

read_timeout

The read_timeout parameter is used to specify how long to wait for incoming data. Default value is 0 which means no time out. It is specified as a duration and it is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    read_timeout: <duration>

shared_key

The shared_key parameter specifies the key that is shared between nodes. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    shared_key: <key>

skip_ingestion_time_on_failure

The skip_ingestion_time_on_failure parameter skips ingestion of the timestamp when the input is broken or an invalid format. It is used with add_ingestion_time. It is specified with a Boolean, the default is false and it is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    add_ingestion_time: true
    skip_ingestion_time_on_failure: true

tag_prefix

YAML Only

The tag_prefix parameter specifies the prefix to append to the Fluentd tag field. This helps in organizing and categorizing the logs as they are ingested. It is specified with a string value and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tag_prefix: <prefix>

tls

The tls parameter is a dictionary type that enables a number of TLS options to be set using sub-parameters.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      <tls options>:

ca_file The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      ca_file: /certs/ca.pem

ca_path The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      ca_path: <path>

client_auth_type The client_auth_type parameter is a child of the tls parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.

The following authentication methods are available:

  • noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
  • requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
  • requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
  • verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
  • requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client
nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      client_auth_type: <auth type>

crt_file The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      crt_file: /certs/server-cert.pem

ignore_certificate_check The ignore_certificate_check parameter is a child of the tls parameter. When set to true, it ignores certificate checks for the remote endpoint. It is specified as a Boolean value and the default is false, indicating that TLS verification will be performed. This is an optional parameter.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      ignore_certificate_check: true

key_file The key_file parameter is a child of the tls parameter. It specifies the private key file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      key_file: /certs/server-key.pem

key_password The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      key_password: <password>

max_version The max_version parameter is a child of the tls parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      max_version: <TLS version>

min_version The min_version parameter is a child of the tls parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
  - name: <node name>
    type: fluentd_input
    port: <port number>
    tls:
      min_version: <TLS version>