Edge Delta Log to Metric Node
5 minute read
Overview
The Log to Metric Node evaluates the body
field for matching patterns and generates metrics.
Example Configuration
Different types of metrics are supported:
Occurrence Count
A simple count of occurrences of logs that match the pattern, for example a count of logs: pattern: "HIGH|high"
.
Numeric Capture
This type matches numeric values in a capture group, for example pattern: " (\\d+)$"
and generates standard metrics: count, min, max, avg, anomaly1 and anomaly2.
Dimension Counter
If named captures in the regex pattern are dimensions, and dimension groups are given, then dimension occurrence stats are generated.
nodes:
- name: http
type: log_to_metric
pattern: "] \"(?P<method>\\w+)"
enabled_stats:
- count
- anomalymin
dimension_groups:
- dimensions: ["method"]
This node will generate an occurrence count
and anomalymin
for each HTTP method:
http_method_get.count
http_method_get.anomalymin
http_method_post.count
http_method_post.anomalymin
etc.
Dimension Numeric Capture
If both dimension and numeric captures are defined in the regex pattern and also in one of the dimension groups, then numeric stats per dimension and per numeric value are generated.
nodes:
- name: "http_request"
type: log_to_metric
pattern: "] \"(?P<method>\\w+) took (?P<latency>\\d+) ms"
interval: 1m
retention: 1h
dimension_groups:
- dimensions: ["method"]
numeric_dimension: "latency"
In this example numeric stats for each HTTP method will be generated such as:
http_request_method_get_latency.avg
http_request_method_get_latency.min
http_request_method_get_latency.max
http_request_method_get_latency.p95
http_request_method_get_latency.p99
http_request_method_get_latency.sum
http_request_method_post_latency.avg
http_request_method_post_latency.min
http_request_method_post_latency.max
http_request_method_post_latency.p95
http_request_method_post_latency.p99
http_request_method_post_latency.sum
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: log_to_metric
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
pattern
The pattern
parameter is used to match log items in the body
field. It is specified as a regex expression and it can include a capture group. If one or more dimension groups are defined, there should be at least one capture group definition. A pattern
is required.
nodes:
- name: <node name>
type: log_to_metric
pattern: <regex pattern>
Optional Parameters
dimension_groups
The dimension_groups
parameter is used to group attributes for metrics. There can be one or more dimension groups. It is specified with child dimensions
elements. It is optional. The dimensions
parameter specifies the names from capture groups that will be used in the metric name or attribute.
It can take a number of options that apply only to that dimension:
custom_suffix
A suffix to append to the metric name.numeric_dimension
The metric value won’t be accepted as 1.0 but rather the value captured from the given dimensionenabled_stats
Statistics to be reported. Valid options are: count, sum, avg, min, max, p25, p75, p95, p99, stddev, anomaly1, anomaly2, anomalymin. The anomalymin option takes min of anomaly1 and anomaly2. This is useful to reduce the alert noise.histogram_limit
The maximum number of histograms per reporter.interval
Interval to report metrics. Default value is 1m.retention
Retention for storing reported metrics to calculate anomaly scores. Default value is 3h.
nodes:
- name: <node name>
type: log_to_metric
pattern: <regex pattern>
dimension_groups:
- dimensions: ["service"]
numeric_dimension: "duration"
custom_suffix: "by_duration"
enabled_stats
The enabled_stats
parameter specifies the statistics that should be reported. It is specified as a list of strings and is optional.
Valid options are:
count
- the number of instances matched.sum
- the sum of matched values.avg
- the average (mean) matching value.min
- the smallest matching value.max
- the largest matching value.p25
- count of values in the 25th percentile.p75
- count of values in the 75th percentile.p95
- count of values in the 95th percentile.p99
- count of values in the 99th percentile.stddev
- the standard deviation.anomaly1
- the proprietary Edge Delta anomaly score 1.anomaly2
- the proprietary Edge Delta anomaly score 2.anomalymin
- the min ofanomaly1
andanomaly2
. This is useful to reduce the alert noise.
The
count
,anomaly1
andanomaly2
metrics are generated for occurrence captures. Whereascount
,min
,max
,avg
,anomaly1
andanomaly2
metrics are generated for numeric captures.
nodes:
- name: <node name>
type: log_to_metric
pattern: <regex pattern>
enabled_stats:
- <statistic type>
- <statistic type>
interval
The interval
parameter specifies the reporting interval for the statistics that the node will generate. It will collect values for the duration of the interval before calculating metrics such as the average. It is specified as a duration and the default is 1 minute. It is optional.
nodes:
- name: <node name>
type: log_to_metric
pattern: <regex pattern>
interval: 2m
metric_name
The metric_name
parameter specifies a custom name for the generated metric. It is specified as a string and the default, if not specified, is to use the node name. It is optional.
nodes:
- name: <node name>
type: log_to_metric
pattern: <regex pattern>
metric_name: <custom name>
retention
The retention
parameter specifies how far back to look to generate anomaly scores. A short retention period will be more sensitive to spikes in metric values. It is specified as a duration and the default is 3 hours. It is optional.
nodes:
- name: <node name>
type: log_to_metric
pattern: <regex pattern>
retention: <duration>