Pipeline Quickstart: Filter Data Items

Add a Telemetry Generator to the Pipeline.

  3 minute read  

Overview

This is the second step in the Pipeline Quickstart guide. In the previous step, you learned how to Add a Telemetry Generator Node.

In this section, you start to configure the processor that is attached to to the Telemetry Generator.

To start, you exclude any log with a low severity keyword in the body:

  • debug
  • info
  • trace
  • notice

1. Open the Processor

Double-click the processor attached to the Telemetry Generator.

The processor configuration page opens.

It shows the live preview of data items coming into this processor in the left pane, and data items leaving the processor in the right pane. In this case, all the data items are logs.

Here is an example of one of the logs:

[Wed Jul 16 02:35:33 2025] [wear:debug] [pid 5239:tid 4666] [client: 205.10.122.184] If we generate the transmitter, we can get to the THX panel through the back-end SDD matrix!

Logs with a low severity keyword in the body are visible (as indicated) in the preview:

2. Add a Filter Processor

Now you start to define the actions that are taken in this node by adding processors. Each processor performs a specific task in order. You add a filter processor to drop specific logs.

  1. Click Add a processor.
  1. Search for filter and select the Filter processor.

Note that Exclude is selected as the action by default. This means that matching items will be excluded.

3. Define Filter Conditions.

  1. Click Add Condition. This is the section that will define which logs to drop.
  1. Next, define the first condition. Set the processor to evaluate the body. Click Path and select Body.
  1. Select matches as the operation. This configures the processor to use a regex match to find the value within the body field.
  1. Enter debug[ as the value. Autocomplete shows a preview of matching logs.

Note addition of the square brackets after the severity text and 1 after trace. This regex context ensures that only the actual severity text is matched, and any mentions of these words in the message, for example, do not create false matches.

In the left pane, logs containing debug] in are marked red to show they are dropped. At the top of the right pane, an 8% decrease in size after processing is predicted.

  1. Click Add nested condition to specify as second condition for the next severity type.

A second condition is added but it has the AND logical operator.

  1. Change the logical operator to OR.
  1. Configure the condition as before, to search the body for matches on info].
  1. To add the next condition, do not add a nested condition, rather select the Add Condition button.
  1. Again, specify an OR logical operator, configure the processor to search the body for matches on trace1.
  1. Lastly, add another condition, specify an OR logical operator, configure the processor to search the body for matches on notice].

4. Save Changes

Click Save on the filter processor.

Note: do not save changes on the whole processor node yet.

At this point, the log volume is estimated to decrease by around 38%.

Next Step

Turn an unstructured log line into structured attributes so later processors can work with discrete fields.