Log to Metric Conversion in Edge Delta

Edge Delta optimizes observability data by converting logs into metrics.

2 minute read

Overview

Log Data Metrics

One way that Edge Delta optimizes observability data is by converting logs into metrics. These metrics can be used for creating dashboards, alerting, and for anomaly detection. Edge Delta agent processors can generate metrics from log lines. For instance, you may be interested to know how many logs with level DEBUG a given service produces, without retaining the entire message:

2022-08-19 17:56:51 DEBUG org.mongodb.driver.cluster - Checking status of cluster0-shard-00-01.mongodb.net:27417

This log can be detected and converted into a metric with the following simple regex processor:

processors:
  regexes:
  - name: debug_level
    pattern: (?)debug

When used in a workflow, the processor will produce a metric debug_level.count which will be reported every minute to the Edge Delta backend as well as 3rd party streaming destinations. In the Edge Delta web app, it can be viewed in the Metrics screen:

debug_level metric viewed in the Edge Delta web app.

The example above is a very simple illustration of using a regex processor to perform a logs to metric conversion. The Edge Delta agent supports numerous processors for generating more sophisticated metrics that can extract:

Numeric value extraction (e.g. latency)
Multi-dimensional values (e.g. response code, method, and latency for an HTTP endpoint)
Top k values (e.g. top 10 endpoints with 5xx status codes)
and several more.

Anomaly Detection

The metrics produced from logs can be further analyzed to detect anomalous values. For more information, see our concept article on anomaly detection.