CEF Pack

This is a pack that allows for processing of CEF logs.

  4 minute read  

Edge Delta Pipeline Pack for CEF (Common Event Format)

Overview

The Edge Delta CEF pack processes logs formatted in the Common Event Format for enhanced monitoring and analysis. It efficiently routes, parses, and transforms log data to make it actionable.

Pack Description

1. Data Ingestion

The process begins with the Pack Source, which acts as the entry point for CEF logs into the pipeline.

2. Route Syslog Entries

Logs are routed through the syslog_router node, a Route node.

- name: syslog_router
  type: route
  paths:
    - path: without_syslog
      condition: regex_match(item["body"], "^CEF:")
      exit_if_matched: true
    - path: with_syslog
      condition: regex_match(item["body"], "CEF:")
      exit_if_matched: true

This node differentiates between logs with and without syslog headers using regex conditions, directing the logs to different paths for tailored processing.

3. Parse Syslog Headers

Logs on the with_syslog path proceed to the with_syslog_parsing node, an ottl_transform node.

- name: with_syslog_parsing
  type: ottl_transform
  statements: |-
    set(cache["grok"], ExtractGrokPatterns(Decode(body, "utf-8"), "%{MONTH:month} %{MONTHDAY:day} %{TIME:time} %{NOTSPACE:host} %{GREEDYDATA:rest}", true))
    set(timestamp, UnixMilli(Time(Concat([cache["grok"]["month"], cache["grok"]["day"], String(Year(Now())), cache["grok"]["time"]], " "), "%h %d %Y %H:%M:%S", "UTC")))
    set(resource["host.name"], cache["grok"]["host"])
    set(attributes["cef_body"], cache["grok"]["rest"])
  • The first OTTL statement uses the ExtractGrokPatterns function to parse the body of the log message after decoding it from UTF-8. It applies a Grok pattern to extract pieces of the message: month, day, time, host, and the rest of the message (using named captures). The extracted values are stored in the cache[“grok”] dictionary for later use.
  • The second statement constructs a timestamp from the extracted month, day, and time fields using the current year (Year(Now())). The Concat function combines these components into a single string with space as the delimiter. The Time function then parses this string according to the specified format to create a timestamp, which is then converted into Unix milliseconds with UnixMilli. The resulting timestamp is stored in the timestamp attribute.
  • The third statement sets the resource[“host.name”] attribute to the host value extracted previously and stored in cache[“grok”][“host”].
  • The last statement assigns the rest of the extracted log message (everything after the host) to the attributes[“cef_body”].

4. Extract CEF Directly

Logs identified as not containing syslog headers go to the without_syslog_parsing ottl_transform node.

- name: without_syslog_parsing
  type: ottl_transform
  statements: |-
    set(attributes["cef_body"], Decode(body, "utf-8"))

This node decodes the content of body from UTF-8 and it is stored in the cef_body attribute.

5. Common CEF Parsing

Both pathways converge at the common_cef_parsing ottl_transform node.

- name: common_cef_parsing
  type: ottl_transform
  statements: |-
    set(cache["cef"], Split(attributes["cef_body"], "|"))
    set(attributes["cef_version"], Int(Substring(cache["cef"][0], 4, 1)))
    set(attributes["cef_device_vendor"], cache["cef"][1])
    set(attributes["cef_device_product"], cache["cef"][2])
    set(attributes["cef_device_version"], cache["cef"][3])
    set(attributes["cef_device_event_class_id"], cache["cef"][4])
    set(attributes["cef_name"], cache["cef"][5])
    set(attributes["cef_severity"], cache["cef"][6])
    set(attributes["cef_attributes"], EDXParseKeyValue(cache["cef"][7], "=", " ", true)) where cache["cef"][7] != ""

This node splits the CEF-formatted body into components, assigning each part to corresponding CEF-specific attributes for comprehensive log analysis:

  1. Splits the cef_body string at each | character. Each resulting segment is stored as an element in the cache["cef"] array, breaking down the CEF message into its constituent parts.
  2. Extracts the CEF version number from the first part of the split message cache["cef"][0]. The Substring function is used to get a specific character range from the string, which is then converted to an integer. The CEF version is stored in attributes["cef_version"].
  3. Stores the device vendor information (second part of the split message) in attributes["cef_device_vendor"].
  4. Stores the device product information (third part of the split message) in attributes["cef_device_product"].
  5. Stores the device version information (fourth part of the split message) in attributes["cef_device_version"].
  6. Stores the event class ID (fifth part of the split message) in attributes["cef_device_event_class_id"].
  7. Stores the event name (sixth part of the split message) in attributes["cef_name"].
  8. Stores the event severity (seventh part of the split message) in attributes["cef_severity"].
  9. Parses any additional key-value pairs present in the eighth part of the split message (if it isn’t empty). EDXParseKeyValue is used to split these key-value pairs on = and separate them by spaces, converting them into a structured format. The parsed key-value data is stored in attributes["cef_attributes"].

6. Cleanup Attributes

Log entries pass through the remove_keys node, a generic_transform node.

- name: remove_keys
  type: generic_transform
  transformations:
    - field_path: item["attributes"]["cef_body"]
      operation: delete

Here, the temporary processing field item["attributes"]["cef_body"] is removed.

7. Output CEF Logs

Finally, processed logs are sent to CEF Logs, a compound_output node, where logs exit the pack, ready for integration and active monitoring.

Sample Input

Jan 18 11:07:53 dsmhost CEF:0|Trend Micro|Deep Security Manager|v0.2.3|600|Administrator Signed In|4|suser=Master

Mar 19 15:19:15 root CEF:0|Trend Micro|Deep Security Agent|v1.10.2-prerelease|123|Out Of Allowed Policy|5|cn1=1

CEF:0|Trend Micro|Deep Security Manager|<DSM version>|600|User Signed In|3|src=10.52.116.160 suser=admin target=admin msg=User signed in from 2001:db8::5

CEF:0|Trend Micro|Deep Security Agent|<DSA version>|4000000|Eicar_test_file|6|cn1=1 cn1Label=Host ID dvchost=hostname cn2=205 cn2Label=Quarantine File Size cs6=ContainerImageName | ContainerName | ContainerID cs6Label=Container filePath=C:\Users\trend\Desktop\eicar.exe act=Delete msg=Realtime TrendMicroDsMalwareTarget=N/A TrendMicroDsMalwareTargetType=N/TrendMicroDsFileMD5=44D88612FEA8A8F36DE82E1278ABB02F TrendMicroDsFileSHA1=3395856CE81F2B7382DEE72602F798B642F14140 TrendMicroDsFileSHA256=275A021BBFB6489E54D471899F7DB9D1663FC695EC2FE2A2C4538AABF651FD0F TrendMicroDsDetectionConfidence=95 TrendMicroDsRelevantDetectionNames=Ransom_CERBER.BZC;Ransom_CERBER.C;Ransom_CRYPNISCA.SM

CEF: 0|Trend Micro|Deep Security Agent|10.2.229|6001200|AppControl detectOnly|6|cn1=202 cn1Label=Host ID dvc=192.168.33.128 TrendMicroDsTenant=Primary TrendMicroDsTenantId=0 fileHash=80D4AC182F97D2AB48EE4310AC51DA5974167C596D133D64A83107B9069745E0 suser=root suid=0 act=detectOnly filePath=/home/user1/Desktop/Directory1//heartbeatSync.sh fsize=20 aggregationType=0 repeatCount=1 cs1=notWhitelisted cs1Label=actionReason cs2=0CC9713BA896193A527213D9C94892D41797EB7C cs2Label=sha1 cs3=7EA8EF10BEB2E9876D4D7F7E5A46CF8D cs3Label=md5