Edge Delta Log to Metric Node

This node is deprecated in favor of the Extract Metric Processor.

Overview

The Log to Metric Node evaluates the body field for matching patterns and generates metrics.

• incoming_data_types: log
• outgoing_data_types: metric

This node requires Edge Delta agent version v0.1.58 or higher.

Example Configuration

Different types of metrics are supported:

• Occurrence Count
• Numeric Capture
• Dimension Counter
• Dimension Numeric Capture
• Field Path Capture

nodes:
- name: occurrence_count
  type: log_to_metric
  pattern: (?i)connection timeout
  interval: 5s
  skip_empty_intervals: false
  only_report_nonzeros: false
  metric_name: connection_timeout
  enabled_stats:
  - count

Required Parameters

name

A descriptive name for the node. This is the name that will appear in pipeline builder and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: log_to_metric

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

pattern

The pattern parameter is used to match log items in the body field. It is specified as a Golang regex expression and it can include a capture group. If one or more dimension groups are defined, there should be at least one capture group definition. A pattern is required.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>

Optional Parameters

anomaly_coefficient

The anomaly_coefficient parameter is used to amplify calculated anomaly scores between 0 and 100. The higher the coefficient the higher the anomaly score will be. It is specified as a float, the default is 10 and is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    anomaly_coefficient: 20

anomaly_confidence_period

The anomaly_confidence_period parameter is used to configure a duration for which to ignore anomalies after discovering a source. This reduces anomaly noise by enabling a baseline. It is specified as duration, the default is 30m and is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    anomaly_confidence_period: 40m

anomaly_tolerance

The anomaly_tolerance parameter is used for handling edge cases for anomaly scores where standard deviation is too small. The default value is 0.01 and it is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    anomaly_tolerance: 0.02

dimension_groups

The dimension_groups parameter is used to group attributes for metrics. There can be one or more dimension groups. It is specified with child dimensions elements. It is optional. The dimensions parameter specifies the names from capture groups that will be used in the metric name or attribute.

It can take a number of options that apply only to that dimension:

• custom_suffix A suffix to append to the metric name.
• numeric_dimension The metric value won’t be accepted as 1.0 but rather the value captured from the given dimension
• enabled_stats Statistics to be reported. Valid options are: count, sum, avg, min, max, p25, p75, p95, p99, stddev, anomaly1, anomaly2, anomalymin. See enabled_stats for the definitions.
• histogram_limit The maximum number of histograms per reporter.
• interval Interval to report metrics. Default value is 1m.
• retention Retention for storing reported metrics to calculate anomaly scores. Default value is 3h.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    dimension_groups:
    - dimensions: ["service"]
      numeric_dimension: "duration"
      custom_suffix: "by_duration"

In addition, there are some YAML only parameters for dimension_groups:

• anomaly_confidence_period Period for anomaly scores to be not reported. Default value is 30m.
• anomaly_tolerance Handles edge cases for anomaly scores where standard deviation is too small. Default value is 0.01.
• anomaly_coefficient Coefficient to amplify calculated anomaly scores between [0, 100] range. Default value is 10.
• skip_empty_intervals When set to true, intervals with no data are skipped. Default is false.
• only_report_nonzeros When set to true, only non-zero statistics are reported. Default is false.
• value_adjustment_rule Contains a mathematical expression to adjust the numeric dimension value.

enabled_stats

The enabled_stats parameter specifies the statistics that should be reported. It is specified as a list of strings and is optional. Valid options are:

• count - the number of instances matched.
• sum - the sum of matched values.
• avg - the average (mean) matching value.
• min - the smallest matching value.
• max - the largest matching value.
• stddev - the standard deviation.
• p25 - count of values in the 25th percentile.
• p50 - count of values in the 50th percentile.
• p75 - count of values in the 75th percentile.
• p95 - count of values in the 95th percentile.
• p99 - count of values in the 99th percentile.
• anomaly1 - represents how many standard deviations a data point is from the mean. Use it for identifying when a metric behaves abnormally based on historical trends. Select Z-score in the Pipeline Builder to include this stat.
• anomaly2 - Uses Inter-Quantile Range to detect outliers based on percentile distribution. Suitable for skewed or non-normal data. Select Inter-quantile range in the Pipeline Builder to include this stat.
• anomalymin - the min of anomaly1 and anomaly2. This is useful to reduce the alert noise. Select Minimum of z-score and IQR in the Pipeline Builder to include this stat.

The count, anomaly1 and anomaly2 metrics are generated for occurrence captures. Whereas count, min, max, avg, anomaly1 and anomaly2 metrics are generated for numeric captures.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    enabled_stats:
    - <statistic type>
    - <statistic type>

field_dimensions

The field_dimensions parameter points to string fields within your payloadDimensions using CEL expressions or bracket notation, such as item[\"attributes\"][\"dimension\"]. This field is useful when working with parsed JSON data. The field_dimensions parameter can be defined alongside the dimensions that come from capture groups in the Golang regex pattern.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    dimension_groups:
    - dimensions:
      - <capture group name>
      field_dimensions:
      - item["resource"]["field.name"]
      field_numeric_dimension: json(item["body"]).details.field

field_numeric_dimension

The field_numeric_dimension parameter defines a numeric field within your payload using a CEL expression or bracket notation, such as item[\"attributes\"][\"numeric_dimension\"]. This field is useful when working with parsed JSON data. You can specify either a numeric dimension or a field numeric dimension, not both.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    dimension_groups:
    - dimensions:
      - <capture group name>
      field_dimensions:
      - item["resource"]["field.name"]
      field_numeric_dimension: json(item["body"]).details.field

group_by

The group_by parameter defines how to aggregate log items based on their properties. Each entry should be an expression (CEL or Go template). When group_by is not set, metrics are grouped by their source. It is specified as a list and is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    group_by:
      - "item._ed.file_path"

interval

The interval parameter specifies the reporting interval for the statistics that the node will generate. It will collect values for the duration of the interval before calculating metrics such as the average. It is specified as a duration and the default is 1 minute. It is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    interval: 2m

Note: Bear in mind the relationship between the interval in the node and the x-axis interval (Roll up) in the Metrics Explorer. An interval of 3 minutes on the node with a 1 minute rollup on the explorer results in no data for two intervals in the explorer followed by an aggregation of the past three minutes. This may be suitable for most use cases but you can make the rollup value equal to or larger than the interval to reduce graph variation and false zero readings. Alternatively you can increase the interval but this will generate more metric traffic.

metric_name

The metric_name parameter specifies a custom name for the generated metric. It is specified as a string and the default, if not specified, is to use the node name. It is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    metric_name: <custom name>

only_report_nonzeros

The only_report_nonzeros parameter configures whether to include statistics that are zero in calculations. It is specified as Boolean, the default is true and is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    only_report_nonzeros: false

retention

The retention parameter specifies how far back to look to generate anomaly scores. A short retention period will be more sensitive to spikes in metric values. It is specified as a duration and the default is 3 hours. It is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    retention: <duration>

skip_empty_intervals

The skip_empty_intervals parameter skips intervals so the anomaly scores are calculated based on history of only non-zero intervals. It is specified with a Boolean, the default value is false and it is optional.

nodes:
  - name: <node name>
    type: log_to_metric
    pattern: <regex pattern>
    skip_empty_intervals: true

value_adjustment_rules

Value adjustment rules define how to modify the value of any numeric capture group as it’s generated. You specify the numeric_dimension capture group to define the value variable, then you provide a mathematical expression that uses the value variable.

nodes:
  - name: log_to_metric
    type: log_to_metric
    pattern: 'error|ERROR|err|ERR service: (?P<service>\w+) duration: (?P<duration>\d+)ms'
    value_adjustment_rules:
      - numeric_dimension: duration
        expression: "value + 200.0"