Log to Metric Conversion
2 minute read
One way that Edge Delta optimizes observability data is by converting logs into metrics. These metrics can be used for creating dashboards, alerting, and for anomaly detection.
Log Data Metrics
Edge Delta agent processors can generate metrics from log lines. For instance, you may be interested to know how many logs with level DEBUG a given service produces, without retaining the entire message:
2022-08-19 17:56:51 DEBUG org.mongodb.driver.cluster - Checking status of cluster0-shard-00-01.mongodb.net:27417
This log can be detected and converted into a metric with the following simple regex processor:
processors: regexes: - name: debug_level pattern: (?)debug
When used in a workflow, the processor will produce a
metric debug_level.count which will be reported every minute to the Edge Delta backend as well as 3rd party streaming destinations. In the Edge Delta web app, it can be viewed in the Metrics screen:
The example above is a very simple illustration of using a regex processor to perform a logs to metric conversion. The Edge Delta agent supports numerous processors for generating more sophisticated metrics that can extract:
- Numeric value extraction (e.g. latency)
- Multi-dimensional values (e.g. response code, method, and latency for an HTTP endpoint)
- Top k values (e.g. top 10 endpoints with 5xx status codes)
- and several more.
The metrics produced from logs can be further analyzed to detect anomalous values. For more information, see our concept article on anomaly detection.