Palo Alto Pack

Edge Delta Pipeline Pack for Palo Alto

Overview

The Edge Delta Palo Alto pack processes logs by parsing, categorizing, and transforming them for better network monitoring and threat analysis. It ensures accurate and structured log entries are routed into appropriate paths for further analysis or processed outputs.

Pack Description

1. Data Ingestion

The data flow begins with the compound_input node, which serves as the entry point into the pack where all logs start their processing journey.

2. Log Type Routing

The logs are directed to the log_type_router node, a Route node. This node evaluates the log body to determine if the log is a TRAFFIC or THREAT type and routes them accordingly.

- name: log_type_router
  type: route
  paths:
    - path: traffic
      condition: regex_match(item["body"], ",TRAFFIC,")
      exit_if_matched: true
    - path: threat
      condition: regex_match(item["body"], ",THREAT,")
      exit_if_matched: true

The routing utilizes regular expression matching to direct logs to the appropriate path based on their entries, ensuring specialized processing for different log types. Any logs that don’t match these conditions are routed to the other_logs pack output.

3.1 Traffic OTTL Transformation

On the traffic path, logs are processed by the traffic_ottl_transform node, an ottl_transform node. This node uses OTTL statements to parse CSV format logs and enriches them with additional properties.

- name: traffic_ottl_transform
  type: ottl_transform
  statements: |-
    // Header string
    set(cache["headers"], "panos.future_use_0,event.receive_timestamp,firewall.serial_number,event.type,event.subtype,panos.future_use_1,event.timestamp,source.ip,destination.ip,source.nat.ip,destination.nat.ip,rule.name,source.user.name,destination.user.name,application,panos.virtual_system,firewall.igress.zone,firewall.egress.zone,firewall.igress.interface.name,firewall.egress.interface.name,panos.log.action,panos.future_use_2,panos.session.id,panos.repeat_count,source.port,destination.port,source.nat.port,destination.nat.port,panos.flags,session.proto,event.outcome,session.bytes,source.bytes,destination.bytes,session.packets,session.start,session.duration,session.url.category,panos.future_use_3,event.sequence_number,panos.action_flags,source.geo,destination.geo,panos.future_use_4,destination.packets,source.packets,panos.session.end_reason,panos.device_group.hierarchy_level_1,panos.device_group.hierarchy_level_2,panos.device_group.hierarchy_level_3,panos.device_group.hierarchy_level_4,panos.virtual_system.name,firewall.host.name,panos.action_source,panos.source.uuid,panos.destination.uuid,panos.tunnel_uuid_imsi,panos.monitor_tag_imei,panos.parent.session.id,panos.parent.session.start_time,panos.tunnel.type,panos.sctp.association_id,panos.sctp.chunks,panos.sctp.chunks.sent,panos.sctp.chunks.received,panos.rule.id,panos.http2_connection,panos.session.link.change_count,panos.policy.id,panos.session.link.switches,panos.sd_wan.cluster,panos.sd_wan.device.type,panos.sd_wan.cluster.type,panos.sd_wan.site,panos.dynamic_user_group.name,panos.xff_address,source.device.category,source.device.profile,source.device.model,source.device.vendor,source.device.os.family,source.device.os.version,source.host.name,source.mac.address,destination.device.category,destination.device.profile,destination.device.model,destination.device.vendor,destination.device.os.family,destination.device.os.version,destination.host.name,destination.mac.address,panos.container_id,panos.pod.namespace,panos.pod.name,source.external_dynamic_list,destination.external_dynamic_list,panos.host.id,panos.user.serial_number,source.dynamic_address_group,destination.dynamic_address_group,session.owner,timestamp.high_resolution,panos.slice.service.type,panos.slice.differentiator,panos.application.subcategory,panos.application.category,panos.application.technology,panos.application.risk,panos.application.characteristic,panos.application.container,panos.tunneled_application,panos.application.saas,panos.application.sanctioned_state,panos.offloaded")
    // Parse CSV
    set(attributes["palo_alto"], ParseCSV(Decode(body, "utf-8"), cache["headers"]))
    // Additional properties
    set(attributes["palo_alto"]["event.category"], "network")
    set(attributes["palo_alto"]["event.type"], ["connection", attributes["palo_alto"]["event.type"]])    

This node is designed to systematically parse and enhance log data, making it more structured and informative for security analysis and network monitoring purposes:

1. set(cache["headers"]...:This statement sets the headers for the CSV parsing by defining a string that includes all the field names expected in the incoming log data. These field names are stored in the cache["headers"] key. This is crucial for the subsequent CSV parsing, as it informs the parser how to interpret each column of the CSV log line. By having a defined header, you ensure that the right pieces of data are accurately mapped to their corresponding attributes.
2. set(attributes["palo_alto"], ParseCSV...: This statement takes the log data contained in the body, decodes it from UTF-8, and then parses it using the ParseCSV function. The cache["headers"] is used here to map the CSV fields correctly to an attributes dictionary under attributes[“palo_alto”]. Parsing the CSV allows for structured access to each piece of log information, enabling more refined analytics and data manipulation downstream.
3. set(attributes["palo_alto"]["event.category"], "network"): This statement assigns the value “network” to the event.category attribute within the parsed Palo Alto attributes. This categorization is useful for filtering and aggregating logs based on their event category in downstream processes or analyses.
4. set(attributes["palo_alto"]["event.type"]...: This statement sets the event.type attribute to an array that includes “connection” and the original event type extracted from the Palo Alto log data. This addition enriches the log details by associating the event with multiple types, which can be instrumental in conducting more nuanced threat assessments or connection analyses.

3.2 Traffic CEL Transformation

Processed traffic logs then flow to the traffic_cel_transform node, a transformation node.

- name: traffic_cel_transform
  type: generic_transform
  transformations:
    - field_path: item["timestamp"]
      operation: upsert
      value: convert_timestamp(item["attributes"]["palo_alto"]["event.timestamp"], "2006-01-02 15:04:05", "Unix Milli")

This node converts event timestamps from strings into Unix milliseconds, ensuring it can be viewed in order in Edge Delta.

3.3 Traffic Logs

Finally, logs on the traffic path are routed to traffic_logs, a pack output, for further processing or storage.

4.1 Threat OTTL Transformation

On the threat path, logs proceed to the threat_ottl_transform node, an ottl_transform node. Similar to traffic logs, this node parses CSV logs and appends relevant event properties.

- name: threat_ottl_transform
  type: ottl_transform
  statements: |-
    // Header string
    set(cache["headers"], "panos.future_use_0,event.receive_timestamp,firewall.serial_number,event.type,event.subtype,panos.future_use_1,event.timestamp,source.ip,destination.ip,source.nat.ip,destination.nat.ip,rule.name,source.user.name,destination.user.name,application,panos.virtual_system,firewall.igress.zone,firewall.egress.zone,firewall.igress.interface.name,firewall.egress.interface.name,panos.log.action,panos.future_use_2,panos.session.id,panos.repeat_count,source.port,destination.port,source.nat.port,destination.nat.port,panos.flags,session.proto,event.outcome,http.url,panos.threat.name,panos.threat.category,panos.threat.severity,panos.threat.direction,panos.sequence_number,panos.action_flags,source.geo,destination.geo,panos.future_use_3,http.content_type,panos.pcap_id,panos.file_digest,panos.cloud,panos.url_index,http.user_agent,panos.file_type,http.forwarded_for,http.referer,email.sender,email.subject,email.recipient,panos.report.id,panos.device_group.hierarchy_level_1,panos.device_group.hierarchy_level_2,panos.device_group.hierarchy_level_3,panos.device_group.hierarchy_level_4,panos.virtual_system.name,firewall.host.name,panos.future_use_4,panos.source.uuid,panos.destination.uuid,http.method,panos.tunnel_uuid_imsi,panos.monitor_tag_imei,panos.parent.session.id,panos.parent.session.start_time,panos.tunnel.type,panos.threat.category,panos.content_version,panos.future_use_5,panos.sctp.association_id,panos.payload_protocol_id,http.headers,panos.url_category_list,panos.rule.id,panos.http2_connection,panos.dynamic_user_group.name,panos.xff_address,source.device.category,source.device.profile,source.device.model,source.device.vendor,source.device.os.family,source.device.os.version,source.host.name,source.mac.address,destination.device.category,destination.device.profile,destination.device.model,destination.device.vendor,destination.device.os.family,destination.device.os.version,destination.host.name,destination.mac.address,panos.container_id,panos.pod.namespace,panos.pod.name,source.external_dynamic_list,destination.external_dynamic_list,panos.host.id,panos.user.serial_number,domain.external_dynamic_list,source.dynamic_address_group,destination.dynamic_address_group,panos.partial_hash,timestamp.high_resolution,panos.reason,panos.justification,panos.slice.service.type,panos.application.subcategory,panos.application.category,panos.application.technology,panos.application.risk,panos.application.characteristic,panos.application.container,panos.tunneled_application,panos.application.saas,panos.application.sanctioned_state")
    // Parse CSV
    set(attributes["palo_alto"], ParseCSV(Decode(body, "utf-8"), cache["headers"]))
    // Additional properties
    set(attributes["palo_alto"]["event.category"], ["network", "threat"])
    set(attributes["palo_alto"]["event.type"], ["connection", "threat", attributes["palo_alto"]["event.type"]])    

Similar to traffic_ottl_transform, this node systematically parses and enriches threat log data, making it structured and informative for cybersecurity monitoring, aiding in threat detection and incident management:

1. set(cache["headers"]...: Similar to traffic_ottl_transform, this statement defines a comprehensive header string containing all expected field names in the incoming threat log data. These field names are stored in cache["headers"]. This definition is essential for instructing the CSV parser on how to interpret the incoming log data by mapping each column of the CSV line to the correct attribute. Proper header definition ensures accurate data extraction and mapping during the parsing process.
2. set(attributes["palo_alto"], ParseCSV...: This statement decodes the incoming log data from UTF-8 format and then parses it using the ParseCSV function. The cache["headers"] is again utilized to guide the parsing, effectively mapping the parsed CSV fields into a structured dictionary under attributes["palo_alto"]. This parsing process standardizes the data format, allowing for easy access and manipulation of structured log fields, which is crucial for detailed threat analysis.
3. set(attributes["palo_alto"]["event.category"]...: This statement enriches the log with the event.category attribute, setting it to an array containing “network” and “threat”. This dual categorization is useful for filtering and analyzing logs based on both network and threat categories, aiding in the identification and prioritization of potential security concerns.
4. set(attributes["palo_alto"]["event.type"]...: This statement sets the event.type attribute to an array that includes “connection”, “threat”, and the original event type derived from the Palo Alto log data. By associating the log entry with multiple types, this statement enhances the log’s detail level, offering richer context for downstream analysis, such as threat detection and security incident response.

4.2 Threat CEL Transformation

Threat logs then advance to the threat_cel_transform node, a transformation node.

- name: threat_cel_transform
  type: generic_transform
  transformations:
    - field_path: item["timestamp"]
      operation: upsert
      value: convert_timestamp(item["attributes"]["palo_alto"]["event.timestamp"], "2006-01-02 15:04:05", "Unix Milli")

This node converts event timestamps from strings into Unix milliseconds, ensuring it can be viewed in order in Edge Delta.

4.3 Threat Logs

Processed logs on the threat path are then routed to threat_logs, an output node for further processing.

5. Other Logs

Logs that don’t match the traffic or threat conditions in the log_type_router node are sent to other_logs, a pack output node.

This completes the data flow of the Palo Alto pack configuration. Each transformation and routing step is carefully designed to parse log data into a structured format, with additional enrichment for network monitoring and threat detection purposes, supporting a robust observability and security framework.

Sample Input

1,2024-11-07 09:43:46,4849102832,TRAFFIC,drop,2,2024-11-07 09:43:58,16.57.196.50,180.152.180.218,9.233.219.172,206.77.190.217,rule-10,Weimann1310,Runte9360,Minkpick,vsys-12,zone-114,zone-7,eth11,eth9,profile-12,3,3114221920,14,44771,63225,33260,58512,149799daa,udp,deny,85099,24068,48033,64,2024-11-07 09:44:32,15,,4,90,1,Taiwan,Croatia,5,18,43,resources-unavailable,21,8,30,1,virtual-system-16,device-14,,29091cf3-eab8-4bfd-9340-a0190a5ce2ea,de9102e9-5f86-4753-86ed-c7641a2c1d01,8500907478671854,55954655457901793,392,2024-11-07 09:45:41,PPTP,447,101,8,19,61d08e18-c543-4a9c-a450-5b83516b3074,227,1,5,,cluster-3,branch,mesh,humannext-generation.name,dynamic-user-group-3,79.122.19.31,category-8,profile-29,some-model-8,"Investormill",MacOS,Ubuntu 24.04.1 LTS,Devonte Waelchi,cc:fc:03:fc:2d:f4,category-8,profile-40,some-model-5,"LoopNet",Linux,11,Juanita Kertzmann,b9:8f:6c:3d:57:77,73388942,namespace-3,palo-alto-84367219,external-dynamic-list-4,external-dynamic-list-10,214379791,37858910,dynamic-address-group-9,dynamic-address-group-16,Torp5739,2024-11-07T09:43:59.008Z,,,observability,networking,browser-based,5,"some,characteristic",container-68327895,Beetry,1,1,0

1,2024-11-07 09:43:46,8411206164,THREAT,start,2,2024-11-07 09:44:04,132.148.254.17,106.238.220.161,22.76.132.107,198.161.65.253,rule-12,Huel2528,Rolfson2503,Metalmix,vsys-3,zone-31,zone-59,eth6,eth11,profile-10,3,1361117620,18,7880,19367,53718,16994,2446c1d4,tcp,reset-server,http://www.principalstreamline.io/robust/supply-chains/implement,26,malware,low,1,98,1,Malawi,Heard Island and McDonald Islands,4,application/book,455,a2cc87ba,forwardvirtual.org,,"Mozilla/5.0 (iPad; CPU OS 8_1_2 like Mac OS X; en-US) AppleWebKit/534.6.1 (KHTML, like Gecko) Version/5.0.5 Mobile/8B118 Safari/6534.6.1",application/x-javascript,146.182.149.11,productunleash.info,andrewalsh@altenwerth.io,"Repudiandae illo qui dolorem.",steviehayes@ziemann.biz,2567578740,20,25,15,3,virtual-system-1,device-5,5,039aebca-8dc4-4db1-94e0-8caaa44c102d,0e5efc94-57e3-4cfa-897e-081cf3b4a8d3,GET,48432952504131,44354862209276425,313,2024-11-07 09:45:21,PPTP,,v0.65.100,6,369,59,nulla=debitis,Adult,ab84d41f-4533-4c4a-bbbc-5ef3197fd16d,322,dynamic-user-group-6,195.109.27.239,category-5,profile-25,some-model-8,"TrialX",MacOS,Ubuntu 24.04.1 LTS,Arlie Hahn,34:de:51:43:93:f7,category-7,profile-31,some-model-4,"ZocDoc",MacOS,Ubuntu 24.04.1 LTS,Erick Oberbrunner,da:2b:10:b8:3e:a7,31145539,namespace-4,palo-alto-57072571,external-dynamic-list-16,external-dynamic-list-8,789643472,899320830,external-dynamic-list-12,dynamic-address-group-7,dynamic-address-group-10,179db978,2024-11-07T09:43:51.008Z,quisquam,dolor,,email,media,browser-based,1,"some,characteristic",container-55394672,Debatealter,1,1

1,2024-11-07 09:43:47,7026153501,TRAFFIC,start,2,2024-11-07 09:43:56,211.235.185.90,149.31.117.33,245.24.5.1,53.17.100.192,rule-4,Pouros6216,Donnelly2643,Jellyfishwake,vsys-16,zone-90,zone-31,eth7,eth7,profile-31,3,4258429502,1,63720,20817,11696,16931,6d32216c,udp,drop ICMP,74495,45928,14381,42,2024-11-07 09:44:32,14,,4,99,1,Micronesia,Jamaica,5,1,13,decrypt-unsupport-param,6,23,2,11,virtual-system-7,device-4,,b2bf27a7-7b7e-43e5-92c4-df6289764293,9e8983b8-5cba-45e6-b3b4-af2f96787f29,1923121451560674,68289432505920440,249,2024-11-07 09:45:27,GRE,16,19,44,44,e3e4eb7c-3d64-41bd-9af7-2b85e2d96380,40,4,3,,cluster-3,hub,mesh,globalsynthesize.info,dynamic-user-group-7,7.129.1.129,category-2,profile-56,some-model-8,"Lenddo",MacOS,11,Katelyn Erdman,61:29:43:c6:74:d8,category-8,profile-10,some-model-2,"Splunk",MacOS,Ubuntu 24.04.1 LTS,Jammie Kuphal,b4:44:00:56:94:8b,59943547,namespace-4,palo-alto-68437336,external-dynamic-list-12,external-dynamic-list-13,314909977,129538294,dynamic-address-group-13,dynamic-address-group-11,Schultz1143,2024-11-07T09:44:11.009Z,,,observability,saas,network-protocol,3,"some,characteristic",container-86151423,Redfile,1,1,1

1,2024-11-07 09:43:47,1671722066,THREAT,deny,2,2024-11-07 09:44:16,166.0.232.249,152.138.162.182,137.25.119.9,119.70.242.135,rule-7,Emmerich3663,Kessler6698,Messagekill,vsys-13,zone-96,zone-98,eth6,eth14,profile-26,3,6275343912,3,924,40645,49217,14525,14a66970a,tcp,override,https://www.investorschemas.io/utilize/experiences,382,phishing,high,0,477,0,China,Belarus,4,application/vnd.nokia.ringing-tone,216,1d2da7c61,corporaterelationships.biz,,"Mozilla/5.0 (X11; Linux i686) AppleWebKit/5331 (KHTML, like Gecko) Chrome/38.0.883.0 Mobile Safari/5331",application/x-pkcs7-mime,207.130.59.88,directmonetize.biz,lauriannerenner@frami.name,"Sed magnam magnam aut at.",wavawisoky@hansen.info,1967770078,11,1,27,31,virtual-system-8,device-1,5,f365ce15-e291-40e9-9f74-a0f98b0ce73d,5d7e438d-d95f-4059-95d8-b3b01ac9a066,GET,5699504249800024,48544600661519269,446,2024-11-07 09:45:49,OpenVPN,,v1.12.156,6,314,48,qui=dolorem,Alcohol and Tobacco,3a27be94-aa00-4506-9e65-bdc4ad52832a,300,dynamic-user-group-8,2.140.183.195,category-4,profile-62,some-model-3,"TuvaLabs",Windows,Ubuntu 24.04.1 LTS,Gia Jaskolski,01:6f:89:5b:0b:f1,category-6,profile-57,some-model-4,"Personal Democracy Media",Linux,15.0.0,Darrion Upton,4d:16:a3:9a:cb:30,27369944,namespace-7,palo-alto-73529923,external-dynamic-list-12,external-dynamic-list-5,820302948,101718031,external-dynamic-list-14,dynamic-address-group-3,dynamic-address-group-3,1e8980568,2024-11-07T09:44:15.009Z,possimus,voluptatem,,observability,networking,network-protocol,2,"some,characteristic",container-62324770,Termitediffer,1,0