ZScaler Pack
6 minute read
Edge Delta Pipeline Pack for ZScaler
Overview
The ZScaler pack processes JSON, KVP, CSV, and TSV logs by routing them through specialized nodes that perform parsing, transformation, and masking functions. Designed to complement ZScaler’s cloud-based security services, this structured approach normalizes logs and enriches them with structured attributes, enhancing their utility for monitoring, analysis, and rapid response in securing enterprise networks.
Pack Description
1. Data Ingestion
The data processing flow begins at the Input node, which is the initial entry point for all logs entering the pack.
2. Routing Logs
Logs are passed to the Route node, which is a critical Route node.
- name: Route
type: route
paths:
- path: zs_nss_dns_json_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_dns_json_log.log"
exit_if_matched: true
- path: zs_nss_tunnel_json_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_tunnel_json_log.log"
exit_if_matched: true
- path: zs_nss_web_json_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_web_json_log.log"
exit_if_matched: true
- path: zs_nss_firewall_kvp_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_firewall_kvp_log.log"
exit_if_matched: true
- path: zs_nss_firewall_json_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_firewall_json_log.log"
exit_if_matched: true
- path: zs_nss_firewall_csv_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_firewall_csv_log.log"
exit_if_matched: true
- path: zs_nss_firewall_tsv_log
condition: item["resource"]["ed.filepath"] == "/var/log/test/zs_nss_firewall_tsv_log.log"
exit_if_matched: true
This node evaluates filename paths to funnel the logs into specific streams based on file path conditions. For instance, it directs logs from /var/log/test/zs_nss_dns_json_log.log to the zs_nss_dns_json_log path and similar such paths for tunnel, web, and firewall logs in different formats (e.g., JSON, KVP, CSV, TSV). These logical routes are established to ensure each file type receives specialized treatment for parsing and transformation tasks downstream.
3. Processing Nodes
3.1 JSON Logs
JSON formatted logs such as DNS, Tunnel, and Web JSON logs are routed to parsing nodes named pja_zs_nss_dns_json, pja_zs_nss_tunnel_json, pja_zs_nss_firewall_json and pja_zs_nss_web_json, respectively.
- name: pja_zs_nss_web_json
type: parse_json_attributes
process_field: item["body"]
field_path: item["attributes"]["PARSED"]
- name: pja_zs_nss_tunnel_json
type: parse_json_attributes
process_field: item["body"]
field_path: item["attributes"]["PARSED"]
- name: pja_zs_nss_dns_json
type: parse_json_attributes
process_field: item["body"]
field_path: item["attributes"]["PARSED"]
- name: pja_zs_nss_firewall_json
type: parse_json_attributes
process_field: item["body"]
field_path: item["attributes"]["PARSED"]
Each of these nodes implements the Parse JSON Attributes functionality. This node type extracts structured attributes from the JSON body field, allowing these attributes to be easily manipulated and analyzed further in the pipeline.
3.2.1 Firewall KVP Logs Comma
For the Key-Value Pair logs, the initial processing step involves mask_zs_nss_firewall_kvp, where tab characters in the data are converted to commas utilizing the Mask node.
- name: mask_zs_nss_firewall_kvp
type: mask
pattern: \t
mask: ','
This transformation step creates a consistent delimiter format, simplifying subsequent parsing.
3.2.1 Firewall KVP Logs Parse
After masking, Firewall KVP logs pass through ottl_zs_nss_firewall_kvp, an OTTL Transform node, where the transformed data is parsed, and key-value pairs are remapped into structured attributes, enhancing their accessibility for analytic applications.
- name: ottl_zs_nss_firewall_kvp
type: ottl_transform
statements: |
set(cache["body"], Decode(body, "utf-8"))
set(attributes["PARSED"], ParseKeyValue(cache["body"],"=",","))
In this node, the Decode function is used to convert an encoded string or byte array (in this case, body) to a decoded string using the specified encoding, which is “utf-8”. See Working with the body. The set function updates the specified telemetry field (cache["body"]) with the decoded value. The second statement parses the string stored in cache["body"] into key-value pairs, using “=” as the key-value delimiter and “,” as the pair delimiter, and assigns the resulting map to attributes["PARSED"]. The ParseKeyValue function extracts key-value pairs from a string, where the key and value are separated by the specified delimiter (here, “=”), and pairs are separated by the specified pair_delimiter (here, “,”). The set function is used to assign the parsed map to attributes["PARSED"].
3.3 Firewall CSV Logs
Firewall logs in CSV format are handled by ottl_zs_nss_firewall_csv, which uses an OTTL Transform node to decode and structure the data through Grok pattern extraction. This process allows specific CSV patterns to be extracted and mapped into a cohesive attribute framework.
- name: ottl_zs_nss_firewall_csv
type: ottl_transform
statements: |+
set(cache["body"],Decode(body, "utf-8"))
set(attributes["PARSED"], ExtractGrokPatterns(cache["body"], "^\"(?P<datetime>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<event_type>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<department>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<locationname>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<cdport>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<csport>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<sdport>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<ssport>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<csip>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<cdip>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<ssip>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<sdip>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<tsip>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<tunsport>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<tuntype>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<action>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<dnat>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<stateful>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<aggregate>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<nwsvc>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<nwapp>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<protocol>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<ipcat>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<destcountry>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<avgduration>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<rulelabel>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<inbytes>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<outbytes>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<duration>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<durationms>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<numsessions>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<ipsrulelabel>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<threatcat>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<devicehostname>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\",\"(?P<deviceowner>[0-9A-Za-z\\_\\ \\-\\:\\.]+)\""))
In this node, the Decode function is used to convert an encoded string or byte array (in this case, body) to a decoded string using the specified encoding, which is “utf-8”. See Working with the body. The set function updates the specified telemetry field (cache["body"]) with the decoded value. In the second statement, the pattern is a sequence of regex expressions with named capture groups (indicated by (?P<name>...)). Each capture group corresponds to a specific field, such as datetime, event_type, department, etc. These capture groups are named so that each extracted piece of data can be directly referenced later. The pattern expects the data to be structured as CSV (Comma-Separated Values) with each field enclosed in quotes. It starts with ^\" which indicates that it should match a quote at the start of the line and captures various data points until the next comma and quote.
3.4.1 Firewall TSV Logs Commas
For the Tab-Separated Value logs, mask_zs_nss_firewall_tsv first converts tabs to commas, making use of a Mask node, effectively aligning with CSV-based processing techniques.
- name: mask_zs_nss_firewall_tsv
type: mask
pattern: \t
mask: ','
3.4.2 Firewall TSV Logs Parse
Following the masking operation, Firewall TSV logs are processed by ottl_zs_nss_firewall_tsv utilizing Grok patterns within an OTTL Transform node, allowing for detailed structured analysis and integration into standardized data sets.
- name: ottl_zs_nss_firewall_tsv
type: ottl_transform
statements: |+
set(cache["body"],Decode(body, "utf-8"))
set(attributes["PARSED"], ExtractGrokPatterns(cache["body"], "^(?P<datetime>[\\w{3}\\s\\w{3}\\s\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\\s\\d{4}]+),(?P<user>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<department>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<locationname>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<cdport>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<csport>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<sdport>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<ssport>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<csip>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<cdip>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<ssip>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<sdip>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<tsip>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<tunsport>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<tuntype>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<action>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<dnat>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<stateful>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<aggregate>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<nwsvc>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<nwapp>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<protocol>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<ipcat>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<destcountry>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<avgduration>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<rulelabel>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<inbytes>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<outbytes>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<duration>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<durationms>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<numsessions>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<ipsrulelabel>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<threatcat>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<devicehostname>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+),(?P<deviceowner>[0-9A-Za-z\\_\\ \\-\\:$@\\.%]+)"))
In this node, the Decode function is used to convert an encoded string or byte array (in this case, body) to a decoded string using the specified encoding, which is “utf-8”. See Working with the body. The set function updates the specified telemetry field (cache["body"]) with the decoded value. In the second statement, the pattern is a regular expression that matches specific components of a log line. The (?P<datetime>...) capture group is designed to extract a datetime field formatted like Mon Jan 01 00:00:00 2000. It uses regex constructs like \\w{3} for three-letter words (e.g., month names) and \\d{2} for two-digit numbers. Other fields such as (?P<user>...), (?P<department>...), etc., follow similar patterns to extract values using [0-9A-Za-z\\_\\ \\-\\:$@\\.%]+ to match a range of alphanumeric characters and special symbols typically found in usernames, department names, and other log components.
4. Data Output
After completing all processing and transformation activities, the refined logs are gathered by the Processed node. This compound output node serves as the conclusive stage in the pack, consolidating logs for further downstream proceesing.
Sample Input
"Mon Feb 24 18:15:11 EST 2025","wilburnbechtelar@mann.biz","Integration","Wiegandstad","Allow","Block","DNS_22","Ipsum ut excepturi sed velit illo.","CNAME","a.internationaltechnologies.com","EMPTY RES","53","11","10.62.161.167","136.129.193.148","category-8","category-2","Credit Karma","districtinnovate.biz"
"Mon Feb 24 18:15:11 EST 2025","gracielabrekke@hermiston.com","HTTP","https://www.forwardengineer.org/expedite/turn-key/e-enable/b2b","Blocked","Amida Technology Solutions","DNS Over HTTPS Services","401243","51949","659","873","Legal Liability","Violence","Violence","Benign","aut","94","molestiae","hic","North Zulauf","Legal Science Partners","6.255.91.143","219.6.201.254","GET","400","Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/5310 (KHTML, like Gecko) Chrome/37.0.840.0 Mobile Safari/5310","corporateviral.biz","File Type Control","URL_Filtering_22","image/gif","Encrypted File","Child Care Desk","nationalbricks-and-clicks.biz","N/A"
"Mon Feb 24 18:15:11 EST 2025","hortensedoyle@harber.com","Branding","Chelseabury","59100","50123","27308","62982","71d7:3857:2499:acfd:3334:263b:7827:473a","7a20:47e8:c58b:637c:7bbd:5bd0:58fc:c8df","cb73:226d:6224:2027:90ae:efec:70c0:25fd","ff4d:3b8b:293f:4818:d2ac:6613:8f70:e83d","bc8c:aa83:1381:9673:e929:29d:5ff1:ab65","28551","Wireshark","Allow","Yes","No","Yes","SSH","Qado Energy, Inc.","DNS","Voluptatum incidunt.","Eritrea","123","Firewall_3","790869","768782","18","517","5","Rule_22","Category_17","Illo ut.","Quandl","seniorincubate.com"
"Mon Feb 24 18:15:11 EST 2025","Tunnel Samples","OpenVPN","VPN_24","Personal, Inc.","146.181.135.97","90.144.138.24","6147","8","46","764823","150684","10","29463"