Azure NSG Flow Pack

Edge Delta Pipeline Pack for Aure NSG Flow Logs

Overview

The Azure NSG Flow Logs pack processes logs by unrolling json, parsing json and creating metrics.

Pack Description

1. Data Ingestion

The process begins with the Pack Source, which acts as the entry point for CEF logs into the pipeline.

2. Unroll JSON

Since the JSON records can be nested under a single log we need to unroll the item.

- name: json_unroll_0137
  type: json_unroll
  user_description: JSON Unroll Processor
  json_field_path: records

This node differentiates between logs with and without syslog headers using regex conditions, directing the logs to different paths for tailored processing.

3. Parse JSON

After unrolling the records into their individual logs the json record will be parsed.

- name: parse_json_attributes_914e
  type: parse_json_attributes
  user_description: Parse JSON Attributes

4. Create metrics

We create metrics to count the number of logs by category.

- name: log_to_metric_9981
    type: log_to_metric
    user_description: Log To Metric
    pattern: .*
    interval: 1m0s
    skip_empty_intervals: false
    only_report_nonzeros: false
    dimension_groups:
      - field_dimensions:
          - item["attributes"]["records"]["category"]
        interval: 1m0s
        skip_empty_intervals: false
        only_report_nonzeros: false

Sample Input

{"records":[{"time":"2018-11-13T12:00:35.3899262Z","systemId":"66aa66aa-bb77-cc88-dd99-00ee00ee00ee","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":2,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF87856","flowTuples":["1542110402,192.0.2.190,10.5.16.4,28746,443,U,I,D,B,,,,","1542110424,203.0.113.10,10.5.16.4,56509,59336,T,I,D,B,,,,","1542110432,198.51.100.8,10.5.16.4,48495,8088,T,I,D,B,,,,"]}]},{"rule":"DefaultRule_AllowInternetOutBound","flows":[{"mac":"000D3AF87856","flowTuples":["1542110377,10.5.16.4,203.0.113.118,59831,443,T,O,A,B,,,,","1542110379,10.5.16.4,203.0.113.117,59932,443,T,O,A,E,1,66,1,66","1542110379,10.5.16.4,203.0.113.115,44931,443,T,O,A,C,30,16978,24,14008","1542110406,10.5.16.4,198.51.100.225,59929,443,T,O,A,E,15,8489,12,7054"]}]}]}},{"time":"2018-11-13T12:01:35.3918317Z","systemId":"66aa66aa-bb77-cc88-dd99-00ee00ee00ee","category":"NetworkSecurityGroupFlowEvent","resourceId":"/SUBSCRIPTIONS/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG","operationName":"NetworkSecurityGroupFlowEvents","properties":{"Version":2,"flows":[{"rule":"DefaultRule_DenyAllInBound","flows":[{"mac":"000D3AF87856","flowTuples":["1542110437,125.64.94.197,10.5.16.4,59752,18264,T,I,D,B,,,,","1542110475,80.211.72.221,10.5.16.4,37433,8088,T,I,D,B,,,,","1542110487,46.101.199.124,10.5.16.4,60577,8088,T,I,D,B,,,,","1542110490,176.119.4.30,10.5.16.4,57067,52801,T,I,D,B,,,,"]}]}]}}]}