Edge Delta Grok Parsing Node

Parse logs as attributes using a Grok parser.

  3 minute read  

Beta

Overview

The Grok parsing node is used for parsing and structuring log data. It relies on a Golang regex pattern to identify and extract specific portions of the log messages and output them as attributes. If no matching patterns are discovered it outputs the log without adding attributes. You use Grok parsing nodes to transform unstructured log data into structured data, making it easier to search, analyze, and visualize.

A number of patterns are provided out of the box from the Knowledge Library, you can specify a custom pattern, or you can use the AI assistant to generate a Grok pattern for you based on a given log sample.

Note: named capture fields can’t have dots, use underscores

  • incoming_data_types: log
  • outgoing_data_types: log

Example Configuration

nodes:
- name: grok
  type: grok
  pattern: clickhouse

Input Log

Consider the following ClickHouse log:

"2024.07.10 06:57:19.511109 [ 8038 ] {} <Info> analytics.module (DataProcessor): Processed data"

Output log

The initial log data is processed and parsed by the Grok node, and each resulting element is added to the attributes filed in the resulting OTEL log message. These items can be referenced as facets in Log Search, accessed as CEL items (such as item["attributes"]["level"]), etc. Note that the original log message is still preserved intact in item["body"].

	{
		"_type": "log",
		"attributes": {
			"level": "Info",
			"message": "analytics.module (DataProcessor): Processed data\"",
			"pid": 8038,
			"timestamp": "2024.07.10 06:57:19.511109",
			"uuid": ""
		},
		"body": "\"2024.07.10 06:57:19.511109 [ 8038 ] {} <Info> analytics.module (DataProcessor): Processed data\"",
		"resource": {
			...
		},
		"timestamp": 1726549251436
	}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: grok

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

pattern

The pattern parameter defines the log pattern that should be used to parse attributes. A Pattern or a Custom Pattern is required.

nodes:
- name: grok
  type: grok
  pattern: clickhouse

Use the Knowledge Library to select a pattern, specify your own a custom pattern, or you use the AI assistant to generate a Grok pattern:

  1. Paste a log sample in the Samples pane
  2. Click Generate a grok pattern for sample logs
  3. A pattern is added to the Pattern field. You can tweak the pattern to replace the capture groups if required.

See Also