Edge Delta EDPorts Input
5 minute read
See the latest versions for HTTP, TCP, and UDP.
Overview
The ed_ports inputs can be used to consume data over tcp or http.
Example
ed_ports:
- labels: "error-counts-per-node"
port: 4545
protocol: http
schema: FlattenedObservation
- labels: "errorcheck"
port: 9000
protocol: tcp
read_size: 10000
read_timeout: 30s
filters:
- source-detection-k8s
- source-detection-custom
- labels: "ed-port-with-auto-detect-line-pattern"
port: 5656
protocol: tcp
auto_detect_line_pattern: true
- labels: "ed-port-with-given-line-pattern"
port: 9091
protocol: http
line_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
late_arrival_handling:
rule_metrics:
ignore_after: 10m
patterns:
ignore_after: 3h
report_with_original_timestamp: true
- labels: "ed-port-with-network-interface"
port: 4545
protocol: tcp
listen: 127.0.0,1
- labels: "ed-port-tcp-with-tls"
port: 4545
protocol: tcp
tls:
crt_file: /certs/server-cert.pem
key_file: /certs/server-key.pem
ca_file: /certs/ca.pem
- labels: "ed-port-https-with-tls"
protocol: https
listen: localhost
port: 443
tls:
crt_file: /certs/server-cert.pem
key_file: /certs/server-key.pem
ca_file: /certs/ca.pem
- labels: "ed-port-with-add-ingestion-time"
port: 4545
protocol: tcp
add_ingestion_time: true
skip_ingestion_time_on_failure: true
Testing an Endpoint
The following commands can be used to test the input:
protocol: http
curl -X POST -d '{"json":"my log"}' <host>:<port><path>
Note: you must specify a path after the port number, such as
/v0/collect.
protocol: tcp
echo "my log" | nc <host> <port>
Required Parameters
labels
Enter a descriptive label for this input. When you create a workflow, you will use this label to enter your input into the workflow.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
port
Enter the port that the agent should listen for.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
protocol
Enter the protocol: tcp, http, or https.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: tcp | http | https
schema: <schema>
read_timeout
Timeout duration for reading from tcp port. It is only applicable for protocol=tcp.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
read_timeout: 30s
schema
Can be empty or FlattenedObservation. If schema is empty then it is assumed the incoming data is raw text lines.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
Optional Parameters
add_ingestion_time
Enter true to ingest a timestamp if the input format is in JSON.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
add_ingestion_time: true
auto_detect_line_pattern
The auto_detect_line_pattern parameter detects line patterns automatically based on the Ragel FSM Based Lexical Recognition process. There is no need to specify line_pattern explicitly. If line_pattern or auto_detect_line_pattern is set then the ingested raw messages should end with “\n” otherwise http type inputs do not work correctly.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
auto_detect_line_pattern: true
enable_incoming_line_anomalies
Enter true to generate anomaly scores.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
enable_incoming_line_anomalies: true
filters
Enter an existing filter to add to this input. To learn more, see Filters. There can be multiple source detectors attached to an input. If one fails (and it is optional) the log will pass the filter and reach the next one. When one of the source detectors can successfully detect the source (meaning all field mappings are satisfied) the rest of the source detectors (if any) will be skipped.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
filters:
- info
line_pattern
The line_pattern parameter is a Golang regex pattern that defines the rule for identifying new lines instead of using New Line ("\n"). If line_pattern or auto_detect_line_pattern is set then the ingested raw messages should end with “\n” otherwise http type inputs do not work correctly.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
line_pattern: '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}'
listen
Enter a network interface where the agent can listen for data.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
listen: <host>
read_size
Read size is only applicable when schema="" and protocol="tcp" otherwise it is not used. If read_size is not provided for the case schema="" and protocol="tcp" then it will be assumed as 1.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
read_size: 10000
skip_ingestion _time_on_failure
Enter true to skip the ingestion of the timestamp when the input is broken or in an invalid format.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
skip_ingestion_time_on_failure: true
tls: ca_file
Enter the absolute path to scan the CA certificate file.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
ca_file: /certs/ca.pem
tls: ca_path
Enter the absolute file path to the CA certificate file.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
ca_path: /var/etc/kafka
tls: client_auth_type
Enter a client authorization type.
You can enter:
- noclientcert
- requestclientcert
- requireanyclientcert
- verifyclientcertifgiven
- requireandverifyclientcert
The default setting is noclientcert.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
client_auth_type:: <auth type>
tls: crt_path
Enter the absolute path to the certificate file.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
crt_file: /certs/server-cert.pem
tls: disable_verify
To disable a TLS verification of a certificate, enter disable_verify: true. To enable a TLS verification of the certificate, you can enter disable_verify: false or you can remove this line entirely.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
disable_verify: true
tls: key_file
Enter the absolute path to the private key file.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
key_file: /certs/server-key.pem
tls: key_password
Enter the password for the key file.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
key_password: <password>
tls: max_version
Enter the maximum version of TLS to accept such as TLSv1_3.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
max_version: <version number>
tls: min_version
Enter the minimum version of TLS to accept such as TLSv1_1.
inputs:
ed_ports:
- labels: "<input name>"
port: <port number>
protocol: <protocol>
schema: <schema>
tls:
min_version: <version number>