Pipeline Quickstart: Normalize Severity

Add a Telemetry Generator to the Pipeline.

  2 minute read  

Overview

This is the fifth step in the Pipeline Quickstart guide. In the previous step, you learned how to Remove unnecessary attributes.

In this section, you normalize the severity classification for logs.

Edge Delta is able to consume top level severity fields and display them as icons in log search, for example. The Parse Severity processor extracts severity fields from data items, maps them to an internal lookup of severity names, and saves them as the root field in the data item called severity_text. The internal map is as follows:

  • fatal
  • error
  • warn
  • info
  • debug
  • trace

1. Add Parse Severity Fields Processor

  1. Click Add a processor.
  1. Select Parse Severity Fields.

A Parse Severity Fields processor is added to the processors stack.

2. Configure Parse Severity Fields Processor

  1. Indicate which attribute contains the log severity. Click Parse from and select attributes severity.

If you close the selected log in the preview pane you can see more of the sample list. Some logs have standard severity values like error or warn, and they now contain the standard icons. But others are not mapped, like emerg, alert or crit.

  1. Next you map non-standard levels in the log attribute to the default Edge Delta values. Click the fatal field and select emerg.
  1. Map the remaining severity levels: add crit to fatal, and alert to error.

Note: The size is decreased by 19%.

  1. Click Save on the Parse Severity Fields processor.

3. Clean Up

Now that the severity text is parsed, select a log in the output field and add a new processor to delete the severity attribute, as per the previous Delete Attributes steps:

This brings the size back to a decrease of about 22%:

  1. Click Save on the Delete processor.

Next Steps

Use the log timestamp as the ingestion timestamp.