Create Metrics from Logs

Create metrics from logs in 5 Minutes.

  3 minute read  

Overview

The Log to Metric Node node transforms log data into quantifiable metrics. This is useful for monitoring, alerting, and reporting purposes. It allows numerical values extracted or generated from logs to be measured, aggregated, and analyzed over time, in a structured format. The transformation involves pattern matching on log entries to generate or extract numeric values and then generating corresponding metrics.

Imagine an e-commerce platform where it’s vital to maintain an optimal user experience. In this scenario you want to configure a Log to Metric node to count 5xx logs.

This is a sample of the generated logs:

Mar 25 17:22:43.190 ed_parallel 188.70.110.238 - - [25/03/2024:15:18:39 +0000] "PUT /utilize HTTP/1.0" 401 79083 "https://www.corporateapplications.com/other/enterprise" "Mozilla/5.0 (Windows 95) AppleWebKit/5311 (KHTML, like Gecko) Chrome/37.0.857.0 Mobile Safari/5311"
Mar 25 17:22:43.190 ed_parallel 207.1.189.215 - - [25/03/2024:15:18:39 +0000] "GET /empower HTTP/1.0" 503 3310 "http://www.forwardclicks-and-mortar.org/platforms/architect/orchestrate" "Mozilla/5.0 (Windows 98; Win 9x 4.90) AppleWebKit/5310 (KHTML, like Gecko) Chrome/37.0.884.0 Mobile Safari/5310"
Mar 25 17:22:43.190 ed_parallel 118.81.25.202 - abernathy5566 [25/03/2024:15:18:39 +0000] "PUT /next-generation HTTP/1.1" 200 28015 "http://www.globalsystems.info/engineer/optimize/vortals/synergize" "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/5311 (KHTML, like Gecko) Chrome/38.0.827.0 Mobile Safari/5311"
Mar 25 17:22:43.190 ed_parallel 160.112.186.178 - - [25/03/2024:15:18:39 +0000] "GET /communities/networks/revolutionary/interactive HTTP/1.0" 501 13828 "https://www.internationalclicks-and-mortar.name/systems/productize/eyeballs" "Opera/8.18 (X11; Linux x86_64; en-US) Presto/2.10.336 Version/13.00"
Mar 25 17:22:43.190 ed_parallel 184.126.183.51 - - [25/03/2024:15:18:39 +0000] "PATCH /magnetic/wireless/paradigms HTTP/2.0" 201 58295 "https://www.chiefdeliverables.io/infrastructures" "Mozilla/5.0 (Windows 98; en-US; rv:1.9.0.20) Gecko/1906-13-11 Firefox/37.0"
Mar 25 17:22:43.190 ed_parallel 168.148.156.147 - - [25/03/2024:15:18:39 +0000] "PATCH /reinvent/cross-media HTTP/1.1" 401 35738 "http://www.nationalevolve.biz/scalable/engage" "Mozilla/5.0 (Macintosh; PPC Mac OS X 10_9_4 rv:2.0) Gecko/1900-18-08 Firefox/37.0"

Prerequisites

You need an Edge Delta account with a Pipeline configuration already created. This is the configuration in which you will create the log to metric configuration.

Configure the Log to Metric Node

Create a Log to Metric node that counts 5xx logs for 1 minute and generates a metric called 5xx_per_minute.

  1. Click Pipelines, select the fleet that handles the NGINX logs and click View/Edit Pipeline.
  2. Click Edit Mode.
  3. Click Add Processor, expand Analytics and select Log to Metric.
  1. Specify a name for the node count_5xx.
  2. Enter the following Golang Regex pattern:
" [5][0-9]{2}
  1. Enter 5xx_per_minute for the metric name.
  1. Click Save Changes.
  2. In this instance, the data source is the demo node. Connect the count_5xx Log to Metric node’s input to the Demo node’s output.

The log to metric node outputs count and anomaly1 metrics by default:

5xx_per_minute.count
5xx_per_minute.anomaly1

You can confirm the metric name and processor logic using the node test feature.

View the Metric

  1. Click Metrics
  2. Ensure that the fleet is selected in the Filter pane.
  3. Select the 5xx_per_minute.count Metric Name.

Next Steps:

Trigger a Metric Alert

See Also

Log to Metric Conversion

Log to Metric Node

Aggregation for Insight

Regex as a String