Checkpoint Pack

This is a checkpoint pack that parses and aggregates Checkpoint logs.

  3 minute read  

Edge Delta Pipeline Pack for Checkpoint

Overview

The Edge Delta Checkpoint pack is designed to process logs by parsing, summarizing, and aggregating them for efficient monitoring and analysis.

Pack Description

1. Data Ingestion

The data flow starts with the Pack Source, serving as the pack’s entry point for all Checkpoint logs. The output is split, or duplicated onto two paths.

2. Parse “:” Structure and Parse “=” Structure

Logs flow into the Parse ":" structure node, a log_transform node that captures structured data from the log body using regex.

- name: Parse structure 1
  type: log_transform
  transformations:
    - field_path: item["attributes"]["structuredData"]
      operation: upsert
      ignore_if_empty: true
      value: regex_capture(item["body"], "^.*?\\[(?P<log>action:.+)\\]").log

This node extracts key-value pairs using regex from the structured part of the log body, helping differentiate actions recorded within logs.

In parallel, logs also pass through the Parse "=" structure node, another log_transform node.

- name: Parse structure 2
  type: log_transform
  transformations:
    - field_path: item["attributes"]["structuredData"]
      operation: upsert
      value: regex_capture(item["body"], "^.*? (?P<log>time=.+)").log

Here, the node extracts key-value pairs separated by equal signs (=) present in log entries, priming them for further processing.

3. Extract Key:Value and Extract Key=Value

Logs are processed by the Extract key:value node, an ottl_transform node that parses key-value pairs on the : path.

- name: Extraction 1
  type: ottl_transform
  statements: merge_maps(attributes, EDXParseKeyValue(attributes["structuredData"], ":", "|", true), "upsert")

This transformation uses OTTL to convert key-value pairs into a usable format, ensuring data is ready for analysis. Logs that cant be transformed are dropped, preventing duplication from the = path.

Similarly, the Extract key=value node, another ottl_transform, processes logs on the = path.

- name: Extraction 2
  type: ottl_transform
  statements: merge_maps(attributes, EDXParseKeyValue(attributes["structuredData"], "=", "|", true), "upsert")

It parses key-value pairs delimited by equals signs, transforming the attributes into a structured data format. Logs that cant be transformed are dropped, preventing duplication from the : path.

4. Update Attributes

Logs are refined in the Update Attributes node, a log_transform node.

- name: Update Attributes
  type: log_transform
  transformations:
    - field_path: item["timestamp"]
      operation: upsert
      ignore_if_empty: true
      value: convert_timestamp(item["attributes"]["time"], "Unix Second", "Unix Milli")
    - field_path: item["attributes"]["structuredData"]
      operation: delete
    - field_path: item["attributes"]["host"]
      operation: upsert
      value: 'has(item.attributes.origin) && item.attributes.origin != "" ? item.attributes.origin : "unknown"'
    - field_path: item["attributes"]["origin"]
      operation: delete
    - field_path: item["attributes"]["time"]
      operation: delete
  • Timestamp Conversion: Converts time attributes into a consistent Unix millisecond format.
  • Structured Data Cleanup: Removes intermediate structured data fields post-extraction.
  • Host Assignment: Sets the host field using the origin, defaulting to “unknown”.

7. Parsed

Finally, processed logs are output via the Parsed node, a compound_output node, exiting the pack for integration with downstream systems or further analysis.

Sample Input

831 <13>1 2023-06-05T15:28:16-04:00 1.1.164.22 time=1685993296|hostname=MIDRBNCPMLM01|product=Firewall|layer_name=Network|layer_uuid=cbba30fa-48d4-418a-8e27-62ab292c4cad|match_id=605|parent_rule=0|rule_action=Accept|rule_name=7.599_._._T2 - - - Wideopen https|rule_uid=eafe8e4c-8014-4f7d-9d0f-d049c00b8148|action=Accept|ifdir=inbound|ifname=bond1.106|logid=0|loguid={0x3b0be45d,0x214545af,0x6505b902,0x2afc45a8}|origin=1.1.160.28|originsicname=REDACTED | user=REDACTED

2281 <13>1 2023-06-05T15:28:16-04:00 1.1.164.22 time=1685993296|hostname=MIDRBNCPMLM01|product=Application - - - Control|app_risk=Unknown|app_risk=Unknown|app_risk=Low|app_category=Business / Economy|app_desc=Google Analytics is a web traffic analytics service from Google. It provides Webmasters with statistics about their website's incoming traffic, such as the traffic's volume and source.|app_id=0|app_id=0|app_id=60340654|app_properties=Business / Economy, Low Risk, Cloud Services|app_sig_id=60340654:3|appi_name=Google Analytics|layer_name=Network|layer_name=Tier1 midrbncplegacyt1_Internet_T1_VS Tier1-outboundweb|layer_name=Tier1-AOB Application|layer_uuid=cbba30fa-48d4-418a-8e27-62ab292c4cad|layer_uuid=dd911585-6593-4c5f-af6f-c66a5747f7a9|layer_uuid=eef9db14-5ccb-42df-a509-e4e2822ef2e1|match_id=148|match_id=16777236|match_id=33554442|matched_category=Business / Economy|parent_rule=0|parent_rule=148|parent_rule=0|rule_action=Inline|rule_action=Accept|rule_action=Accept|rule_name=7.142_._._|rule_name=Wideopen http/https|rule_name=10_._._|rule_uid=77e1bc39-395a-4bbe-905f-2319ed2d1925|rule_uid=51eaf8f7-dc93-4788-b699-d5542c624c1e|rule_uid=19e3cf2f-f52c-4cdd-94fb-8cd444fe7a41|app_risk=Low|app_category=Business / Economy|app_desc=Google Analytics is a web traffic analytics service from Google. It provides Webmasters with statistics about their website's incoming traffic, such as the traffic's volume and source.|app_id=60340654|app_properties=Business / Economy, Low Risk, Cloud Services|app_sig_id=60340654:3|appi_name=Google Analytics|matched_category=Business / Economy|action=Accept|conn_direction=Outgoing|ifdir=inbound|ifname=bond0.596|logid=256|loguid={0x29ba252a,0x2287cb4d,0xcd643955,0x9693dd5b}|origin=1.1.204.18|originsicname=REDACTED | user=REDACTED

702 <13>1 2023-06-05T15:28:16-04:00 1.1.164.22 time=1685993296|hostname=MIDRBNCPMLM01|product=HTTPS - - - Inspection|action=HTTPS Bypass|ifdir=inbound|ifname=eth2-01|loguid={0x647e3750,0x44,0xda44ecc,0x27444c62}|origin=10.226.37.5|originsicname=REDACTED | user=REDACTED

781 <13>1 2023-06-05T15:28:16-04:00 1.1.164.22 time=1685993296|hostname=MIDRBNCPMLM01|product=Log - - - Update|action=Accept|contextnum=1|ifdir=inbound|ifname=bond0.596|logid=6|loguid={0x25347ad5,0x8fe25aeb,0xfeab4c93,0x8a1df7ad}|origin=1.1.204.18|originsicname=REDACTED | start_time=1685993286

760 <13>1 2023-06-05T15:28:16-04:00 1.1.164.22 time=1685993296|hostname=MIDRBNCPMLM01|product=URL - - - Filtering|action=Accept|conn_direction=Outgoing|ifdir=inbound|ifname=bond0.499|logid=288|loguid={0x4b39b568,0xf54688ee,0x959a3246,0xccda007a}|origin=1.1.204.18|originsicname=REDACTED | update_count=2

660 <13>1 2023-06-05T15:28:17-04:00 1.1.164.22 time=1685989515|hostname=MIDRBNCPMLM01|product=Firewall|action=Decrypt|ifdir=inbound|ifname=WAN|loguid={0x348328e4,0x4def7c85,0x534be638,0x196ee01a}|origin=66.51.152.162|time=1685989515|version=1|community=MegaPort_123NET|dst=10.124.203.10|fw_subproduct=VPN-1|inzone=External|methods - - - =ESP: AES-256 + SHA256|origin_sic_name=REDACTED | vpn_feature_name=VPN