Edge Delta Grok Parsing Node

Overview

The Grok parsing node is used for parsing and structuring log data. It relies on a Golang regex pattern to identify and extract specific portions of the log messages and output them as attributes. If no matching patterns are discovered it outputs the log without adding attributes. You use Grok parsing nodes to transform unstructured log data into structured data, making it easier to search, analyze, and visualize.

A number of patterns are provided out of the box from the Knowledge Library. Alternatively you can specify a custom pattern.

Note: named capture fields can’t have dots, use underscores

• incoming_data_types: log
• outgoing_data_types: log

Example Configuration

nodes:
- name: grok
  type: grok
  pattern: clickhouse

Input Log

Consider the following ClickHouse log:

"2024.07.10 06:57:19.511109 [ 8038 ] {} <Info> analytics.module (DataProcessor): Processed data"

Output log

The initial log data is processed and parsed by the Grok node, and each resulting element is added to the attributes filed in the resulting OTEL log message. These items can be referenced as facets in Log Search, accessed as CEL items (such as item["attributes"]["level"]), etc. Note that the original log message is still preserved intact in item["body"].

{
  "_type": "log"
  "attributes": {
    "level": "Info"
    "message": "analytics.module (DataProcessor): Processed data"
    "pid": 8038
    "timestamp": "2024.07.10 06:57:19.511109"
    "uuid": ""
  }
  "body": "2024.07.10 06:57:19.511109 [ 8038 ] {} <Info> analytics.module (DataProcessor): Processed data"
"
  "resource": { • • • }
  "timestamp": 1720594639511
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: grok

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

pattern

The pattern parameter defines the log pattern that should be used to parse attributes.

• nginx
• clickhouse
• vpb
• coredns
• redis
• apache_common
• apache_error
• and apache_combined
• AWS VPC Flow
• MYSQL Slow Query
• MYSQL Error

A Pattern or a Custom Pattern is required.

nodes:
- name: grok
  type: grok
  pattern: clickhouse

Optional Parameters

custom_pattern

The custom_pattern parameter defines a regex pattern for a log format that should be used to parse attributes. A Pattern or a Custom Pattern is required.

nodes:
- name: custom_processor
  type: grok
  custom_pattern: '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'

See Also

• Knowledge Library