Edge Delta Sample Processor

The Edge Delta sample processor takes a representative sample of incoming data items.

3 minute read

The sample processor enables you to filter and manage logs and traces based on a given sampling type and percentage, using consistent probabilistic sampling. It works by letting a specified percentage of data pass through based on various fields, and provides additional configurable options to suit different criteria for sampling.

For detailed instructions on how to use multiprocessors, see Use Multiprocessors.

Configuration

In this example, 50% of logs in the input pane have been dropped without consideration of their contents.

Options

Select a telemetry type

You can specify, log, metric, trace or all. It is specified using the interface, which generates a YAML list item for you under the data_types parameter. This defines the data item types against which the processor must operate. If data_types is not specified, the default value is all. It is optional.

It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    data_types:
    - log

condition

The condition parameter contains a conditional phrase of an OTTL statement. It restricts operation of the processor to only data items where the condition is met. Those data items that do not match the condition are passed without processing. You configure it in the interface and an OTTL condition is generated. It is optional. You can select one of the following operators:

Operator	Name	Description	Example
`==`	Equal to	Returns `true` if both values are exactly the same	`attributes["status"] == "OK"`
`!=`	Not equal to	Returns `true` if the values are not the same	`attributes["level"] != "debug"`
`>`	Greater than	Returns `true` if the left value is greater than the right	`attributes["duration_ms"] > 1000`
`>=`	Greater than or equal	Returns `true` if the left value is greater than or equal to the right	`attributes["score"] >= 90`
`<`	Less than	Returns `true` if the left value is less than the right	`attributes["load"] < 0.75`
`<=`	Less than or equal	Returns `true` if the left value is less than or equal to the right	`attributes["retries"] <= 3`
`matches`	Regex match	Returns `true` if the string matches a regular expression	`isMatch(attributes["name"], ".*\\.name$"`

It is defined in YAML as follows:

- name: _multiprocessor
  type: sequence
  processors:
  - type: <processor type>
    condition: attributes["request"]["path"] == "/json/view"

Pass through on failure

This option determines whether items should pass through if an error occurs during the evaluation of sampling.

Percentage

This option specifies the percentage of items that will be allowed to pass through the node without filtering.

Timestamp granularity

This duration option specifies the granularity of timestamps when sampling by timestamp, with a minimum allowed granularity of 1 millisecond.

Priority field

This option defines a field whose presence will override the default sampling percentage if the field has a value.

Field paths

This option lists the paths to fields used for determining how sampling should occur. If not specified, traces are sampled by trace ID and logs by timestamp, service name, and body.

Final

The final parameter specifies whether successfully processed data items should continue to subsequent processors within the same multiprocessor node. Data items that fail to be processed by the processor will be passed to the next processor in the node regardless of this setting. You select the slider in the tool which specifies it for you in the YAML as a Boolean. The default is false and it is optional.

It is defined in YAML as follows:

- name: multiprocessor
  type: sequence
  processors:
    - type: <processor type>
    final: true