Edge Delta Log to Metric Node

Extract metrics from logs using the Edge Delta Log to Metric Node.

  5 minute read  

Overview

The Log to Metric Node evaluates the body field for matching patterns and generates metrics.

Example Configuration

Different types of metrics are supported:

Occurrence Count

A simple count of occurrences of logs that match the pattern, for example a count of logs: pattern: "HIGH|high".

Numeric Capture

This type matches numeric values in a capture group, for example pattern: " (\\d+)$" and generates standard metrics: count, min, max, avg, anomaly1 and anomaly2.

Dimension Counter

If named captures in the regex pattern are dimensions, and dimension groups are given, then dimension occurrence stats are generated.

nodes:
  - name: http
    type: log_to_metric
    pattern: "] \"(?P<method>\\w+)"
    enabled_stats: 
    - count
    - anomalymin
    dimension_groups:
    - dimensions: ["method"]

This node will generate an occurrence count and anomalymin for each HTTP method:

  • http_method_get.count
  • http_method_get.anomalymin
  • http_method_post.count
  • http_method_post.anomalymin etc.

Dimension Numeric Capture

If both dimension and numeric captures are defined in the regex pattern and also in one of the dimension groups, then numeric stats per dimension and per numeric value are generated.

nodes:
  - name: "http_request"
    type: log_to_metric
    pattern: "] \"(?P<method>\\w+) took (?P<latency>\\d+) ms"
    interval: 1m
    retention: 1h
    dimension_groups:
    - dimensions: ["method"]
      numeric_dimension: "latency"

In this example numeric stats for each HTTP method will be generated such as:

  • http_request_method_get_latency.avg
  • http_request_method_get_latency.min
  • http_request_method_get_latency.max
  • http_request_method_get_latency.p95
  • http_request_method_get_latency.p99
  • http_request_method_get_latency.sum
  • http_request_method_post_latency.avg
  • http_request_method_post_latency.min
  • http_request_method_post_latency.max
  • http_request_method_post_latency.p95
  • http_request_method_post_latency.p99
  • http_request_method_post_latency.sum

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: log_to_metric

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

pattern

The pattern parameter is used to match log items in the body field. It is specified as a regex expression and it can include a capture group. If one or more dimension groups are defined, there should be at least one capture group definition. A pattern is required.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>

Optional Parameters

dimension_groups

The dimension_groups parameter is used to group attributes for metrics. There can be one or more dimension groups. It is specified with child dimensions elements. It is optional. The dimensions parameter specifies the names from capture groups that will be used in the metric name or attribute.

It can take a number of options that apply only to that dimension:

  • custom_suffix A suffix to append to the metric name.
  • numeric_dimension The metric value won’t be accepted as 1.0 but rather the value captured from the given dimension
  • enabled_stats Statistics to be reported. Valid options are: count, sum, avg, min, max, p25, p75, p95, p99, stddev, anomaly1, anomaly2, anomalymin. The anomalymin option takes min of anomaly1 and anomaly2. This is useful to reduce the alert noise.
  • histogram_limit The maximum number of histograms per reporter.
  • interval Interval to report metrics. Default value is 1m.
  • retention Retention for storing reported metrics to calculate anomaly scores. Default value is 3h.
nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    dimension_groups:
    - dimensions: ["service"]
      numeric_dimension: "duration"
      custom_suffix: "by_duration"

enabled_stats

The enabled_stats parameter specifies the statistics that should be reported. It is specified as a list of strings and is optional. Valid options are:

  • count - the number of instances matched.
  • sum - the sum of matched values.
  • avg - the average (mean) matching value.
  • min - the smallest matching value.
  • max - the largest matching value.
  • p25 - count of values in the 25th percentile.
  • p75 - count of values in the 75th percentile.
  • p95 - count of values in the 95th percentile.
  • p99 - count of values in the 99th percentile.
  • stddev - the standard deviation.
  • anomaly1 - the proprietary Edge Delta anomaly score 1.
  • anomaly2 - the proprietary Edge Delta anomaly score 2.
  • anomalymin - the min of anomaly1 and anomaly2. This is useful to reduce the alert noise.

The count, anomaly1 and anomaly2 metrics are generated for occurrence captures. Whereas count, min, max, avg, anomaly1 and anomaly2 metrics are generated for numeric captures.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    enabled_stats: 
    - <statistic type>
    - <statistic type>

interval

The interval parameter specifies the reporting interval for the statistics that the node will generate. It will collect values for the duration of the interval before calculating metrics such as the average. It is specified as a duration and the default is 1 minute. It is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    interval: 2m

metric_name

The metric_name parameter specifies a custom name for the generated metric. It is specified as a string and the default, if not specified, is to use the node name. It is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    metric_name: <custom name>

retention

The retention parameter specifies how far back to look to generate anomaly scores. A short retention period will be more sensitive to spikes in metric values. It is specified as a duration and the default is 3 hours. It is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    retention: <duration>