Edge Delta Grok Parsing Node
3 minute read
Overview
The Grok parsing node is used for parsing and structuring log data. It relies on a Golang regex pattern to identify and extract specific portions of the log messages and output them as attributes. If no matching patterns are discovered it outputs the log without adding attributes. You use Grok parsing nodes to transform unstructured log data into structured data, making it easier to search, analyze, and visualize.
A number of patterns are provided out of the box from the Knowledge Library. Alternatively you can specify a custom pattern.
Note: named capture fields can’t have dots, use underscores
![](/images/grok-node-form.png)
Example Configuration
nodes:
- name: grok
type: grok
pattern: clickhouse
Input Log
Consider the following ClickHouse log:
"2024.07.10 06:57:19.511109 [ 8038 ] {} <Info> analytics.module (DataProcessor): Processed data"
Output log
The initial log data is processed and parsed by the Grok node, and each resulting element is added to the attributes
filed in the resulting OTEL log message. These items can be referenced as facets in Log Search, accessed as CEL items (such as item["attributes"]["level"]
), etc. Note that the original log message is still preserved intact in item["body"]
.
{
"_type": "log"
"attributes": {
"level": "Info"
"message": "analytics.module (DataProcessor): Processed data"
"pid": 8038
"timestamp": "2024.07.10 06:57:19.511109"
"uuid": ""
}
"body": "2024.07.10 06:57:19.511109 [ 8038 ] {} <Info> analytics.module (DataProcessor): Processed data"
"
"resource": { • • • }
"timestamp": 1720594639511
}
Required Parameters
name
A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a -
and a space followed by the string. It is a required parameter for all nodes.
nodes:
- name: <node name>
type: <node type>
type: grok
The type
parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.
nodes:
- name: <node name>
type: <node type>
pattern
The pattern parameter defines the log pattern that should be used to parse attributes.
- nginx
- clickhouse
- vpb
- coredns
- redis
- apache_common
- apache_error
- and apache_combined
- AWS VPC Flow
- MYSQL Slow Query
- MYSQL Error
A Pattern or a Custom Pattern is required.
nodes:
- name: grok
type: grok
pattern: clickhouse
Optional Parameters
custom_pattern
The custom_pattern
parameter defines a regex pattern for a log format that should be used to parse attributes. A Pattern or a Custom Pattern is required.
nodes:
- name: custom_processor
type: grok
custom_pattern: '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'