Processors Overview
  • Dark
    Light

Processors Overview

  • Dark
    Light

Overview

A processor performs logs to metrics conversions of incoming raw log data. Once configured, the processor will populate the Anomalies and Insights pages as well as the Metrics view. 

Note

To learn more about Anomalies, see Anomalies.

To learn more about Insights, see Insights.


Review Supported Processor Types

The Edge Delta App supports the following processor types:

Processor TypeDescription

Cluster

This processor type finds patterns in logs, and then groups (or clusters) these patterns based on similarities.

Regex

Regex processors match data in logs using a regular expression. They perform any required logic on the matches such as counting instances or determining dimensions and averages for numeric pattern matches. They can contain trigger thresholds to send alerts. Finally, they send metrics to get reported in the web app.

Ratio

This processor takes one successful regex pattern and one failed regex pattern to calculate a success ratio.

Top-K

This processor monitors top K records, such as k=10, where the records are identified with one or more named regex group values combined together.

Trace

This processor is useful to track events that have a unique ID and clear start and end logs.


Create and Manage a Processor

To create and manage a processor, you must populate a YAML file.

To access the YAML file for a new configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
  2. Click Create Configuration.
  3. Click YAML.
  4. Enter your desired parameters, and then click Save.

To access the YAML file for an existing configuration:

  1. In the Edge Delta App, on the left-side navigation, click Data Pipeline, and then click Agent Settings.
  2. Locate the desired configuration, then under Actions, click the vertical ellipses, and then click Edit.
  3. Review the YAML file, make your changes, and then click Save.

Learn about Clustered Invariants

You can use this section to learn how Edge Delta calculates similarities between invariants for clustering purposes.

When a new log passes through the pipeline:

  • Variants are identified via a proprietary Ragel FSM-based tokenization process
  • The identified variants are stripped from the log and replaced with wildcards
  • The remaining invariant components are compared to existing pattern sets to calculate similarities
    • Invariant components are calculated for similarities so that the invariants can be transformed and clustered into structured log messages.

There are 2 ways to calculate similarities:

  • Drain
  • Levenshtein distance

Learn about Drain

Drain is the default log parsing algorithm used to cluster logs. This algorithm is based on a parse tree, with a fixed depth to guide the log group search process. This workflow helps to avoid a deep and unbalanced tree.

When a new raw log message arrives, Edge Delta processes the message with the Ragel FSM-based tokenization process. Then, Edge Delta searches for a log group through the nodes of the tree, based on the token prefix.

If a suitable log group is found, then Edge Delta will also calculates the similarities between the log message and the log event stored in the log group. If the similarity rate is above a certain threshold, then the log message will be matched with the log event stored in that log group.

  • If not, a new log group will be created based on the log message.

To accelerate this process, Edge Delta designs a parse tree with a fixed depth, and nodes with fixed children to guide the log group search. This helps to limit the number of log groups that a raw log message needs to be compared to.

Since Edge Delta uses a drain log parse tree for clustering based on a common prefix, Edge Delta can easily merge the clusters by using their ancestors in the tree. The merge level determines how many levels Edge Delta will go up in the tree.

image.png

Learn about Levenshtein Distance

Levenshtein distance is a string metric that measures the difference between 2 sequences. The Levenshtein distance between 2 words is the minimum number of single-character edits (insertions, deletions, or substitutions) required to change one word into the other.

image.png

When a new raw log message arrives, Edge Delta processes the message with the Ragel FSM-based tokenization process.

Then, Edge Delta uses the Levenshtein distance algorithm to calculate similarities between tokens. If there is a similarity above a certain threshold, then Edge Delta will determine that these logs belong to the same log group.

The similarity calculation is based on the minimum number of operations required to make 2 tokens the same. If the required operation number is below a certain threshold, then the 2 tokens are more similar and grouped in the same log group. Otherwise, a new log group will be created based on the log message.


Was this article helpful?

What's Next
Changing your password will log you out immediately. Use the new password to log back in.
First name must have atleast 2 characters. Numbers and special characters are not allowed.
Last name must have atleast 1 characters. Numbers and special characters are not allowed.
Enter a valid email
Enter a valid password
Your profile has been successfully updated.