Edge Delta EDPorts Input

Ports and protocols for the agent to listen for incoming traffic.

See the latest versions for HTTP, TCP, and UDP.

Overview

The ed_ports inputs can be used to consume data over tcp or http.

Example

 ed_ports:
    - labels: "error-counts-per-node"
      port: 4545
      protocol: http
      schema: FlattenedObservation
    - labels: "errorcheck"
      port: 9000
      protocol: tcp
      read_size: 10000
      read_timeout: 30s
      filters:
      - source-detection-k8s
      - source-detection-custom
    - labels: "ed-port-with-auto-detect-line-pattern"
      port: 5656
      protocol: tcp
      auto_detect_line_pattern: true
    - labels: "ed-port-with-given-line-pattern"
      port: 9091
      protocol: http
      line_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
      late_arrival_handling:
        rule_metrics:
          ignore_after: 10m
        patterns:
          ignore_after: 3h
          report_with_original_timestamp: true
    - labels: "ed-port-with-network-interface"
      port: 4545
      protocol: tcp
      listen: 127.0.0,1
    - labels: "ed-port-tcp-with-tls"
      port: 4545
      protocol: tcp
      tls:
        crt_file: /certs/server-cert.pem
        key_file: /certs/server-key.pem
        ca_file: /certs/ca.pem
    - labels: "ed-port-https-with-tls"
      protocol: https
      listen: localhost
      port: 443
      tls:
        crt_file: /certs/server-cert.pem
        key_file: /certs/server-key.pem
        ca_file: /certs/ca.pem
    - labels: "ed-port-with-add-ingestion-time"
      port: 4545
      protocol: tcp
      add_ingestion_time: true
      skip_ingestion_time_on_failure: true 

Testing an Endpoint

The following commands can be used to test the input:

protocol: http

curl -X POST -d '{"json":"my log"}' <host>:<port><path>

Note: you must specify a path after the port number, such as /v0/collect.

protocol: tcp

echo "my log" | nc <host> <port>

Required Parameters

labels

Enter a descriptive label for this input. When you create a workflow, you will use this label to enter your input into the workflow.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>

port

Enter the port that the agent should listen for.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>

protocol

Enter the protocol: tcp, http, or https.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: tcp | http | https
      schema: <schema>

read_timeout

Timeout duration for reading from tcp port. It is only applicable for protocol=tcp.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      read_timeout: 30s

schema

Can be empty or FlattenedObservation. If schema is empty then it is assumed the incoming data is raw text lines.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>

Optional Parameters

add_ingestion_time

Enter true to ingest a timestamp if the input format is in JSON.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      add_ingestion_time: true

auto_detect_line_pattern

The auto_detect_line_pattern parameter detects line patterns automatically based on the Ragel FSM Based Lexical Recognition process. There is no need to specify line_pattern explicitly. If line_pattern or auto_detect_line_pattern is set then the ingested raw messages should end with “\n” otherwise http type inputs do not work correctly.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      auto_detect_line_pattern: true

enable_incoming_line_anomalies

Enter true to generate anomaly scores.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      enable_incoming_line_anomalies: true

filters

Enter an existing filter to add to this input. To learn more, see Filters. There can be multiple source detectors attached to an input. If one fails (and it is optional) the log will pass the filter and reach the next one. When one of the source detectors can successfully detect the source (meaning all field mappings are satisfied) the rest of the source detectors (if any) will be skipped.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      filters: 
        - info 

line_pattern

The line_pattern parameter is a Golang regex pattern that defines the rule for identifying new lines instead of using New Line ("\n"). If line_pattern or auto_detect_line_pattern is set then the ingested raw messages should end with “\n” otherwise http type inputs do not work correctly.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      line_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'

listen

Enter a network interface where the agent can listen for data.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      listen: <host>

read_size

Read size is only applicable when schema="" and protocol="tcp" otherwise it is not used. If read_size is not provided for the case schema="" and protocol="tcp" then it will be assumed as 1.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      read_size: 10000

skip_ingestion _time_on_failure

Enter true to skip the ingestion of the timestamp when the input is broken or in an invalid format.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      skip_ingestion_time_on_failure: true

tls: ca_file

Enter the absolute path to scan the CA certificate file.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        ca_file: /certs/ca.pem

tls: ca_path

Enter the absolute file path to the CA certificate file.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        ca_path: /var/etc/kafka

tls: client_auth_type

Enter a client authorization type.

You can enter:

  • noclientcert
  • requestclientcert
  • requireanyclientcert
  • verifyclientcertifgiven
  • requireandverifyclientcert

The default setting is noclientcert.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        client_auth_type:: <auth type>

tls: crt_path

Enter the absolute path to the certificate file.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        crt_file: /certs/server-cert.pem

tls: disable_verify

To disable a TLS verification of a certificate, enter disable_verify: true. To enable a TLS verification of the certificate, you can enter disable_verify: false or you can remove this line entirely.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        disable_verify: true

tls: key_file

Enter the absolute path to the private key file.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        key_file: /certs/server-key.pem

tls: key_password

Enter the password for the key file.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        key_password: <password>

tls: max_version

Enter the maximum version of TLS to accept such as TLSv1_3.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        max_version: <version number>

tls: min_version

Enter the minimum version of TLS to accept such as TLSv1_1.

inputs:
  ed_ports:
    - labels: "<input name>"
      port: <port number>
      protocol: <protocol>
      schema: <schema>
      tls:
        min_version: <version number>