Edge Delta Parse JSON Node

Parse JSON logs as attributes.

  2 minute read  

Beta

Overview

The Parse JSON node parses fully structured JSON logs as attributes.

Custom Attributes

Custom attributes are flattened to create key value pairs of type (string, string), for example:

"attributes": {
  "a": 1
  "b": {
    "bb": "cc"
  }  
}

These attributes will be flattened as follows:

"attributes": {
  "a": "1",
  "b.bb": "cc"
}

Note: In the case of conflicts, such as this example, the behavior is non-deterministic:

"attributes": {
  "a.b": "foo"
  "a": {
    "b": "bar"
  }
}

Example Configuration

nodes:
  - name: parse_json
    type: parse_json_attributes
    process_field: item["body"]

Example Input Log

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": `{"id":"64f019a3bdd0a1e54d254c53","currentJob":{"title":"Developer","salary":"mask;"},"jobs":[{"title":"CEO","salary":"R$ 6.875,45"},{"title":"medic","salary":"R$ 0.884,24"}]}`,
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "container.name": "echo:latest",
    "k8s.controller.kind": "Deployment",
    "k8s.controller.logical.name": "api-deployment",
  },
  "_type": "log",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "k8s_labels": {
      "app": "edgedelta",
    }
  },  
  "id": "475121c4-e0b4-4669-a7f6-783fa13348ce",
}

Example Output Log

{
  "timestamp": "1581452773000000789",
  "severity_text": "Error",
  "body": `{"id":"64f019a3bdd0a1e54d254c53","currentJob":{"title":"Developer","salary":"mask;"},"jobs":[{"title":"CEO","salary":"R$ 6.875,45"},{"title":"medic","salary":"R$ 0.884,24"}]}`,
  "resource": {
    "host.name": "host-1",
    "ed.tag": "test-config",
    "ed.org.id": "0481a213-....",
    "__src_name": "source_name",
    "src_type": "K8s",
    "__logical_source": "logical_source",
    "__short_src_name": "short_source_name",
    "__group_name": "group_name",
    "k8s.namespace.name": "edgedelta",
    "k8s.pod.name": "api-deployment-d79fab72249c",
    "k8s.container.name": "echo:latest",
    "k8s.controller.kind": "Deployment",
    "k8s.controller.logical.name": "api-deployment",
  },
  "_type": "log",
  "attributes": {
    "pod_id":"api-deployment-d79fab72249c-vtq9x", // user enrichment
    "instance_id":"i-1234567890abcdef0",  // user enrichment
    "instance_name":"test-name",  // user enrichment
    "currentJob": {
        "salary": "mask;",
        "title": "Developer"
    },
    "id": "64f019a3bdd0a1e54d254c53",
    "jobs": [
        {
            "salary": "R$ 6.875,45",
            "title": "CEO"
        },
        {
            "salary": "R$ 0.884,24",
            "title": "medic"
        }
    ],
    "k8s_labels": {
      "app": "edgedelta",
    }
  },  
  "id": "475121c4-e0b4-4669-a7f6-783fa13348ce",
}

Required Parameters

name

A descriptive name for the node. This is the name that will appear in Visual Pipelines and you can reference this node in the yaml using the name. It must be unique across all nodes. It is a yaml list element so it begins with a - and a space followed by the string. It is a required parameter for all nodes.

nodes:
  - name: <node name>
    type: <node type>

type: parse_json_attributes

The type parameter specifies the type of node being configured. It is specified as a string from a closed list of node types. It is a required parameter.

nodes:
  - name: <node name>
    type: <node type>

Optional Parameters

process_field

The process_field parameter defines a dot separated field path on which to apply parsing. It is specified as a string and is optional. The default is item["body"].

nodes:
  - name: <node name>
    type: parse_json_attributes
    process_field: item["body"]

A dot separated field path that describes the location within the payload where the operation should be applied.