Microsoft DNS Server Pack

Edge Delta Pipeline Pack for Microsoft DNS

Overview

The Edge Delta Microsoft DNS pack processes logs by transforming and normalizing them for consistent analysis and tracking of DNS activities.

Pack Description

1. Data Ingestion

The data flow starts with the compound_input node as the entry point into the pack.

- name: compound_input
  type: compound_input

2. Log Transformation

Logs are processed by the transform node, which is an OTTL Transform node.

- name: transform
  type: ottl_transform
  statements: |-
    // Decode body
    set(cache["parsed_body"], Decode(body, "utf-8"))
    // Extract Grok Pattern
    set(cache["grok_result"], ExtractGrokPatterns(cache["parsed_body"], "(?P<date>\d{1,2}-\d{1,2}-\d{4} \d{1,2}:\d{2}:\d{2} (AM|PM))\s+%{NOTSPACE:threadID}\s+%{NOTSPACE:context}\s+%{NOTSPACE:packetID}\s+%{NOTSPACE:protocol}\s+%{NOTSPACE:direction}\s+%{NOTSPACE:remoteIP}\s+%{NOTSPACE:transactionID} %{DATA:packetType}\s+%{NOTSPACE:opcode}\s+\[%{NOTSPACE:flagsInHex}\s+%{NOTSPACE:flagsInCharArray}\s+%{NOTSPACE:responseCode}\]\s+%{NOTSPACE:questionType}\s+%{NOTSPACE:questionName}", true))
    // Question Name Normalization
    // 1) Change number groups into dots
    replace_pattern(cache["grok_result"]["questionName"], "\(\d+\)", ".")
    // 2) Remove first dot
    replace_pattern(cache["grok_result"]["questionName"], "^\.", "")
    // Packet Type Normalization
    set(cache["grok_result"]["packetType"], "Query") where IsMatch(cache["grok_result"]["packetType"], "\s+") or cache["grok_result"]["packetType"] == nil
    set(cache["grok_result"]["questionName"], "Response") where IsMatch(cache["grok_result"]["questionName"], "R")
    // Update Time
    set(timestamp, UnixMilli(Time(cache["grok_result"]["date"], "%q-%g-%Y %l:%M:%S %p")))
    // Update Attributes
    set(attributes["fields"], cache["grok_result"])    

• Body Decoding: The Decode(body, "utf-8") operation decodes the log message body using UTF-8. See Working with the body and Working with a Cache.
• Pattern Extraction: ExtractGrokPatterns() is used for Grok pattern extraction. This function parses unstructured data into a structured format by extracting named capture groups from the target string using a Grok pattern. If no matches are found, it returns an empty map.
• Normalization: Changes the questionName and packetType to a more consistent format. It replaces all string sections that match a regex pattern with a new value. In this case, it is used to normalize the questionName by replacing number groups with dots and removing the first dot.
• Time Normalization: Converts date strings into UNIX milliseconds. See Manage Log Timestamps.
  • The IsMatch function checks if a target string matches a regex pattern. It returns true if a match is found and false otherwise. Here, it is used to conditionally set values for packetType and questionName based on their content.
  • The UnixMilli function converts a time.Time object to its Unix timestamp representation, measured in milliseconds since January 1, 1970 UTC. It is applied to a time value obtained from the Time function to set the timestamp.
  • The Time function parses a string representation of time into a time.Time object based on a specified format. It is used here to parse the date extracted by the Grok pattern.
• Attributes Update: The parsed and normalized attributes are set in the attributes field for further usage.

3. Log Output

After transformation, logs flow to the parsed_logs compound output node.

- name: parsed_logs
  type: compound_output

The logs that do not fit into defined patterns (or fail the transformation) are redirected to the other_logs compound output node.

- name: other_logs
  type: compound_output

Sample Input

1-7-2025 4:47:53 PM	8DD4	PACKET	AE90D1E389923ACF	UDP	Rcv	211.180.29.192	9d5f R	?	[a53e	D	REFUSED]	MX	(7)exuding(18)centralopen-source(3)com(0)

1-7-2025 4:47:47 PM	AB71	PACKET	323E8D15DFD495E3	UDP	Snd	192.206.158.167	4d7d 	U	[e578	DR	ACCEPTED]	MX	(10)reciprocal(18)seniorleading-edge(3)org(0)

1-7-2025 4:47:41 PM	A87C	PACKET	635308F4F63DD0B0	TCP	Snd	253.74.1.95	6b67 	Q	[d5e1	TD	REFUSED]	A	(12)programmable(16)internalgranular(3)org(0)

1-7-2025 4:47:30 PM	4338	PACKET	9ED5BB2D1AA8B1DE	TCP	Rcv	135.101.16.164	62cb R	N	[1449	R	REFUSED]	MX	(11)application(10)lead24-365(4)name(0)