Log to Metric Conversion in Edge Delta
2 minute read
Log Data Metrics
One way that Edge Delta optimizes observability data is by converting logs into metrics. These metrics can be used for creating dashboards, alerting, and for anomaly detection. Edge Delta agent processors can generate metrics from log lines. For instance, you may be interested to know how many logs with level DEBUG a given service produces, without retaining the entire message:
2022-08-19 17:56:51 DEBUG org.mongodb.driver.cluster - Checking status of cluster0-shard-00-01.mongodb.net:27417
This log can be detected and converted into a metric using the log to metric node with (?)debug configured for the pattern.
The processor will produce a metric debug_level.count which will be reported every minute to the Edge Delta backend as well as 3rd party streaming destinations. In the Edge Delta web app, it can be viewed in the Metrics Explorer.
This example is an illustration of using a regex processor to perform a logs to metric conversion. The Edge Delta agent supports numerous nodes for generating more sophisticated metrics that can extract:
- Numeric value extraction (e.g. latency)
- Multi-dimensional values (e.g. response code, method, and latency for an HTTP endpoint)
- Top k values (e.g. top 10 endpoints with 5xx status codes)
- and several more.
Anomaly Detection
The metrics produced from logs can be further analyzed to detect anomalous values. For more information, see anomaly detection.