Edge Delta Extract JSON Field Node

Overview

The extract_json_field node extracts values from JSON log content in a particular field and uses it as the log’s body field. This node is useful for focusing log analysis on specific pieces of data or for simplifying complex log documents into a more manageable form that retains only the essential information required for further processing and analysis.

• incoming_data_types: log
• outgoing_data_types: log

Example Configuration

- name: extract_json_test
  type: extract_json_field
  field_path: records[0].data
  keep_log_if_failed: true

In this example the [0] index indicates that we want to extract the data field from the first object in the records array.

Input Log Body:

{
    "timestamp": "2023-04-23T13:00:00.000Z",
    "logLevel": "INFO",
    "service": "DataService",
    "nodeId": "node9",
    "records": [
        {
            "data": {
                "user": "user789",
                "action": "data_fetch",
                "details": "Fetched 42 records"
            }
        }
    ]
}

Output Log Body:

{
  "user": "user789",
  "action": "data_fetch",
  "details": "Fetched 42 records"
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the YAML using the name. It must be unique across all nodes. It is a YAML list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: extract_json_field

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

field_path

The field_path parameter is a dot separated path of the field. It captures the JSON path from each log item. It supports for wildcards * which is useful for capturing all values in an array. It is specified as a string and is required.

nodes:
  - name: <node name>
    type: extract_json_field
    field_path: "records.[*]"

Optional Parameters

keep_log_if_failed

The keep_log_if_failed parameter defines whether to drop items with processor failures such as improper JSON documents or non-existing JSON paths. It is specified as a Boolean and the default is false, indicating that failures will be dropped. It is optional.

nodes:
  - name: <node name>
    type: extract_json_field
    field_path: "records.[*]"
    keep_log_if_failed: true