Log Patterns

The Edge Delta agent uses a proprietary algorithm to detect repeated patterns in log messages.

  2 minute read  

Pattern Detection

The Edge Delta agent uses a proprietary algorithm to automatically detect repeated patterns in log messages. This allows it to optimize data by reporting patterns and their frequency rather than streaming the full log messages. Variant values within a pattern are expressed as a wildcard (*).

Automatically detected log patterns, with count and other per-pattern statistics.

Cluster Processors

Log patterns are detected by processors that are configured on an Edge Delta agent. The cluster processor will track and report the most frequently occurring patterns for the default or specified interval, and it will send the specified number of full log samples for each pattern. You configure pattern detection by adding a cluster processor to a workflow.

A cluster processor configuration.

Sentiment Analysis

Every pattern detected by the Edge Delta agent is further analyzed to check for negative sentiment. Negative sentiment is determined by checking for the presence of specific keywords in the pattern (e.g. error, exception, fail, etc.). Some keywords such as debug are considered neutralizing because they automatically offset negative keyword matches in the pattern.

The negative and neutralizing keywords used in sentiment analysis can be configured in the Pipeline Settings for an account. They are applied to all agents within the account.

Default negative and neutralizing keywords.

Pattern Visualization

Log patterns detected by the Edge Delta agent can be viewed in the Edge Delta web app as well as 3rd party observability tools.

Edge Delta Web App

Log patterns can be viewed in multiple places in the Edge Delta Web app:
The Patterns screen shows a breakdown of Negative Patterns (all patterns with negative sentiment).

Negative Patterns chart, showing patterns with negative sentiment.

All Patterns shows patterns with any sentiment.

All patterns chart (with negative or neutral sentiment)

All patterns also shows a tabular view of the most frequently occurring patterns and their associated statistics.

All patterns table, with statistics including sentiment.

The Kubernetes Overview shows patterns for a particular cluster, namespace, controller, or container image.

Viewing a namespace in Kubernetes view, including negative patterns (grouped by controller).

3rd Party Tools

Patterns can also be viewed in any streaming destination that accepts log data.

Log Pattern visualization in Datadog.
Log Pattern visualization in Sumo Logic.

Anomaly Detection in Log Patterns

Once pattern data is sent to the Edge Delta backend, it can be further analyzed for anomalies. See our article on anomaly detection for more details.