Default Edge Delta Configuration
3 minute read
Overview
When you create a new agent configuration, it includes a default pipeline configuration.
Input
Kubernetes Input
The default pipeline starts with a Kubernetes input. It is configured to ingest logs from certain Kubernetes resources, and not others. See more documentation on configuring a Kubernetes input here.
The node ingests logs from all namespaces in the cluster (see the include field). Of those, it excludes the following resources using the exclude field:
k8s.namespace.name=kube-systemk8s.namespace.name=kube-publick8s.namespace.name=kube-node-leasek8s.pod.name=edgedeltak8s.pod.name=prometheusk8s.pod.name=promtailk8s.pod.name=node-exporter
Processors
Mask Node
The Kubernetes input node is connected to a Mask node named mask_ssn using a link. It is configured with a regex pattern that identifies US social security numbers and obfuscates them with he word REDACTED.
See more documentation on configuring a mask node here.
The mask_ssn node connects to a number of downstream processor nodes arranged in parallel. The parallel construction of the pipeline indicates that all output from mask_ssn is sent to each downstream processor i.e. the data is duplicated per link.
For conditional routing along parallel paths, where data is only sent down one of the paths based on its characteristics, you can use a Route node.
Regex Filter
A Regex filter node named drop_trace_level is one of the processors fed by the mask_ssn node in the default configuration.
It is configured to identify logs containing the string TRACE. The default behavior of a regex filter is to identify and pass only those logs that match the specified pattern. However, in this configuration the behavior is inverted by setting the Negate switch to True. Now the node will drop only those logs containing TRACE anywhere in the log, and it will pass all other logs. In the default configuration, the drop_trace_level node sends logs to the ed_archive_output, and they can be accessed using Log Search.
Log to Metrics
There are three log to metrics nodes that are fed logs from the mask_ssn node in the default configuration:
error_monitoring
The error_monitoring log to metric node is configured to create metrics based on logs containing the Golang regex pattern (?i)error. This node will capture any logs containing the text error or any case variation.
It outputs sum and count metrics that you can view in the metrics explorer.
exception_monitoring
The exception_monitoring log to metric node is configured to create metrics based on logs containing the Golang regex pattern (?i)exception. This node will capture any logs containing the text exception or any case variation.
It outputs sum and count metrics that you can view in the metrics explorer.
negative_sentiment_monitoring
The negative_sentiment_monitoring log to metric node is configured to create metrics based on logs containing the Golang regex pattern (?i)(exception|fail|timeout|broken|caught|denied|abort|insufficient|killed|killing|malformed|unsuccessful|outofmemory|panic|undefined). This node will capture any logs containing any of the negative sentiment keywords or any case variation of them.
It outputs sum and count metrics that you can view in the metrics explorer.
See more documentation on configuring a log to metric node here.
Log to Pattern
A log to pattern node named log_to_pattern is connected downstream to the mask_ssn node in the default configuration. It reports every minute to the ed_patterns output node with any patterns and samples detected on the edge.
Patterns and samples detected by this node on the edge, as well as any detected in post-processing on the backend, can be explored on the Logs - Patterns page.
Outputs
The default configuration sends data to three Edge Delta outputs:
ed_archiveis used to send archive data to the Edge Delta SaaS to power Log Search.ed_metricsis used to send metrics to the Edge Delta SaaS to power the metrics explorer page as well Kubernetes Overview, Pipeline Status, filters and others. In addition it collects the agent version, agent heartbeats, and last activated data.ed_patternis used to send pattern data to the Edge Delta SaaS to power the Logs - Patterns page.