Edge Delta HTTP Source

Consume HTTP log traffic.

  6 minute read  

Overview

The HTTP source node directly receives log data from applications that transmit logs over HTTP, which is a common method for centralized log collection, especially in microservice architectures and event-driven architectures.

Note: Customers with HTTP source, TCP source or UDP source nodes should not use or update to Agent version v0.1.97.

  • outgoing_data_types: log

Example Configuration

nodes:
- name: my_http_input
  type: http_input
  port: 3421
  read_timeout: 10s
  included_paths:
    - /v1/.* 
  authentication:
    strategy: Bearer      
    secret: "testXYZ"

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: http_input

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

port

Enter the port number that the http_input type node should listen on. It is specified as an integer and is a required parameter.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>

Optional Parameters

included_paths

The included_paths parameter is used to filter out traffic that is not from the specified path. This helps reduce input noise. It is specified as one or more dictionary entries and you can include wildcards. It is an optional parameter.

nodes:
- name: http_input
  type: http_input
  port: 3421
  read_timeout: 10s
  included_paths:
    - /v1/.*

authentication

The authentication parameter defines the type of authentication. It is specified as a string. Basic and Bearer are supported. For Bearer authentication you specify a secret token. For Basic, you specify the username and password. Authentication is an optional parameter.

Bearer:

nodes:
- name: http_input
  type: http_input
  port: 3421
  read_timeout: 10s
  included_paths:
    - /v1/.* 
  authentication:
    strategy: Bearer      
    secret: "<your_bearer_token>"

Basic:

- name: my_http_input
  type: http_input
  port: 8080
  included_paths:
    - /v1/.* 
  authentication:
    strategy: Basic      
    username: <username>    
    password: <password>

listen

The listen parameter is used to specify the address to listen to for incoming traffic. It is specified as a string and it is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    listen: <host>

read_timeout

The read_timeout parameter is used to specify how long to wait for incoming data. Default value is 0 which means no time out. It is specified as a duration and it is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    read_timeout: 10s

tls

The tls parameter is a dictionary type that enables a number of options to be set using sub-parameters.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      <tls options>:

ca_file The ca_file parameter is a child of the tls parameter. It specifies the CA certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      ca_file: /certs/ca.pem

ca_path The ca_path parameter is a child of the tls parameter. It specifies the location of the CA certificate files. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      ca_path: /var/etc/kafka

client_auth_type The client_auth_type parameter is a child of the tls parameter. It specifies the authentication type to use for the connection. It is specified as a string from a closed list and is optional.

The following authentication methods are available:

  • noclientcert indicates that no client certificate should be requested during the handshake, and if any certificates are sent they will not be verified.
  • requestclientcert indicates that a client certificate should be requested during the handshake, but does not require that the client send any certificates.
  • requireanyclientcert indicates that a client certificate should be requested during the handshake, and that at least one certificate is required from the client, but that certificate is not required to be valid.
  • verifyclientcertifgiven indicates that a client certificate should be requested during the handshake, but does not require that the client sends a certificate. If the client does send a certificate it is required to be valid.
  • requireandverifyclientcert indicates that a client certificate should be requested during the handshake, and that at least one valid certificate is required to be sent by the client
nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      client_auth_type: <auth type>

crt_file The crt_file parameter is a child of the tls parameter. It specifies the certificate file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      crt_file: /certs/server-cert.pem

ignore_certificate_check The ignore_certificate_check parameter is a child of the tls parameter. When set to true, it ignores certificate checks for the remote endpoint. It is specified as a Boolean value and the default is false, indicating that TLS verification will be performed. This is an optional parameter.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      ignore_certificate_check: true

key_file The key_file parameter is a child of the tls parameter. It specifies the key file. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      key_file: /certs/server-key.pem

key_password The key_password parameter is a child of the tls parameter. It specifies the key password. When the private key_file location is provided, this file can also be provided to get the password of the private key. It is specified as a string and is optional.

nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      key_password: <password>

max_version The max_version parameter is a child of the tls parameter. It specifies the maximum version of TLS to accept. It is specified as a string and is optional.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      max_version: <TLS version>

min_version The min_version parameter is a child of the tls parameter. It specifies the minimum version of TLS to accept. It is specified as a string and is optional. The default is TLSv1_2.

You can select one of the following options:

  • TLSv1_0
  • TLSv1_1
  • TLSv1_2
  • TLSv1_3
nodes:
  - name: <node name>
    type: http_input
    port: <port number>
    tls:
      min_version: <TLS version>

Testing an Endpoint

The following command can be used to test the input:

curl -X POST -d '{"json":"my log"}' <host>:<port><path>

Note: you must specify a path after the port number, such as /v0/collect.