Edge Delta Ratio Processor

Processors Recap

You can configure a processor to perform logs to metrics conversions of incoming raw log data. Once configured, the processor will populate the Anomalies and Insights pages as well as the Metrics view. See Processors for more information about processors. Edge Delta has a number of processor types, one of which is a ratio processor.

Ratio Processors

A ratio processor monitors the ratio of specific success patterns against failure patterns. Each of these patterns is defined with a Golang regular expression. The ratio is determined as follows:

ratio = success / (success+failure)

The processor can report anomalies in the success ratio and it can be configured with trigger thresholds to issue alerts.

To manage a ratio processor, you configure the agent yaml with processor parameters in a ratios section.

processor:
  ratios:
    - name: <processor_name>
      success_pattern: <regex_pattern_success>
      failure_pattern: <regex_pattern_fail>
      <optional_parameter>: <parameter_value>

Ratio Processor Example

There are no ratio processors configured by default. You can design a ratio processor by identifying a ratio of success and failure metrics that you want to track.

In the following example, the ratio of “request succeeded” messages to “request failed” messages is monitored and reported on. They are specified in the success_pattern and failure_pattern parameters. The processor will only examine the parts of the logs allowed by the filter named message, which has been specified in the filters section of the agent yaml. The reference to that filter is configured with the filters parameter. The ratio is calculated after gathering metrics for a 2 minute interval. In addition, an alert is triggered if the ratio value scores an anomaly probability of more than 50 for two reporting intervals in a row, configured with the anomaly_probability_percentage and consecutive trigger parameters. To determine if a ratio is anomalous the processor will compare it to the ratios for the previous 4 hours, configured with the retention parameter.

processors:
  ratios:
    - name: request-error-ratio
      success_pattern: "request succeeded"
      failure_pattern: "request failed"
      interval: 2m
      retention: 4h
      trigger_thresholds:
        anomaly_probability_percentage: 50
        consecutive: 2
      filters: 
        - message

Required Parameters

name

The name parameter specifies a name for the ratio processor. You refer to this name in other places, for example to refer to a specific processor in a workflow. Names must be unique within the processor section. It is a yaml list element so it begins with a - and a space followed by the string. A name is a required parameter for a ratio processor.

processors:
  ratios:
    - name: <processor-name>
      success_pattern: <regex_pattern>
      failure_pattern: <regex_pattern> 

success_pattern

The success_pattern parameter specifies a Golang regex pattern that the ratio processor will look for, the count of which will be used as the success portion of the ratio. It is a string that should be wrapped in quotes to handle escapes. A success_pattern is a required parameter for a ratio processor.

processors:
  ratios:
    - name: <processor_name>
      success_pattern: <regex_pattern> 
      failure_pattern: <regex_pattern> 

failure_pattern

The failure_pattern parameter specifies a Golang regex pattern that the ratio processor will look for, the count of which will be used as the failure portion of the ratio. It is a string that should be wrapped in quotes to handle escapes. A failure_pattern is a required parameter for a ratio processor.

processors:
  ratios:
    - name: <processor_name>
      success_pattern: <regex_pattern> 
      failure_pattern: <regex_pattern> 

Optional Parameters

filters

The filters parameter refers to a defined filter that has been configured in the filters section of the agent yaml. The filter contains logic that defines where in the log to apply the processor. All other data is ignored by the processor. You can use a filter to prevent the processor from processing portions of a log that contain sensitive data. Filters are a yaml list element so they begin with a - and a space. They are defined with a string that matches a filter name.

processors:
  ratios:
    - name: <processor_name>
      success_pattern: <regex_pattern> 
      failure_pattern: <regex_pattern> 
      filters:
        - <filter_reference>

interval

The interval parameter specifies the reporting interval for the statistics that a ratio processor will generate. A processor will collect values for the duration of the interval before calculating metrics such as the ratio. The default is 1 minute. It is specified in the Golang duration format.

processors:
  ratios:
    - name: <processor_name>
      success_pattern: <regex_pattern> 
      failure_pattern: <regex_pattern> 
      interval: <duration>

retention

The retention parameter specifies how far back to look when the ratio processor generates anomaly scores. A short retention period will be more sensitive to spikes in metric values. The default for a processor is 3 hours. It is specified as a Golang duration.

processors:
  ratios:
    - name: <processor_name>
      success_pattern: <regex_pattern> 
      failure_pattern: <regex_pattern> 
      retention: <duration>

trigger_thresholds

The trigger_thresholds parameter is a dictionary type that can specify certain child parameters with specific combinations of thresholds. When a threshold is reached a trigger destination (specified in the corresponding workflow) is notified.

processors:
  <processor type>:
    - name: <processor_name>
      pattern: <regex_pattern> 
      trigger_thresholds:
        <trigger_threshold_parameter>: <integer>

The following thresholds can be configured for ratio processors:

anomaly_probability_percentage The anomaly_probability_percentage parameter sets the threshold for a trigger based on the Edge Delta agent’s confidence that an event is an anomaly. The range is 0-100 where 100 is the highest confidence that an event is an anomaly. There is no default value. It is configured as an integer.
upper_limit_per_interval The upper_limit_per_interval parameter sets the maximum number of events within the reporting interval. A higher occurrence would trigger a notification for too many events. It is configured as an integer.
lower_limit_per_interval The lower_limit_per_interval parameter sets the minimum number of events within the reporting interval. A lower occurrence would trigger a notification for not enough events. It is configured as an integer. consecutive The consecutive parameter sets the minimum number of times a threshold must be triggered before an alert is issued. It requires another trigger_threshold parameter to be set for the processor. The default is zero. It is configured as an integer.