Pipeline Quickstart: Create Structured Attributes

Add a Telemetry Generator to the Pipeline.

  2 minute read  

Overview

This is the third step in the Pipeline Quickstart guide. In the previous step, you learned how to Reduce noise and volume by dropping lower‑value severities (debug, info, trace, notice).

In this section, you extract data from the body and use them as structured attributes. To start, examine a log by selecting it in either preview pane.

There are potential attributes of interest for downstream processing either in the pipeline or later in the backend:

  • log_timestamp: the whole RFC 822‑style date string
  • module: component name
  • severity: raw severity text
  • client_ip: source IP

A Grok parsing processor can be used to extract these fields from the body in a single operation.

1. Add Grok Parsing Processor

  1. Click Add a processor.
  2. Search for and select Parse Grok.

A Grok parser is added to the processor stack under the Filter processor. This indicates that the Filter operation will occur first.

2. Configure the Grok Pattern

Enter a Grok pattern that captures the required fields. You can use the Grok library to assist in designing the pattern, or use an AI assistant. In this instance, enter the following pattern in the Grok pattern field:

\[%{DATA:log_timestamp}\] \[%{WORD:module}:%{WORD:severity}\] \[pid %{NUMBER:pid}:tid %{NUMBER:tid}\] \[client: %{IP:client_ip}\]

Note Grok requires every parsed segment of the pattern to include at least one named capture group, so the [pid ...:tid ...] section must extract pid and tid as fields, even though they are not required, or the entire pattern fails silently. Realistically, separate Parse Regex processors might be more efficient, but Grok is used in this demonstration to illustrate deletion of fields later.

The live preview output pane shows that attributes have been successfully added to the logs.

  1. Click Save on the Grok processor.

Note: The post processing size is now estimated to have decreased by only 19%. It shows the combined effect both the Filter and the Grok processor. While Filter reduced data size, Grok increased it slightly.

Next Step

Remove unnecessary attributes.