Cloudflare Pack

Edge Delta Pipeline Pack for Cloudflare

Overview

The Edge Delta Cloudflare pack is designed to efficiently parse, suppress, and transform logs for improved monitoring and actionable insights. The pack uses a predefined sequence of nodes to manage Cloudflare log data effectively.

Pack Description

1. Data Ingestion

The data flow initiates with the Source node as the entry point into the pack, where logs begin their processing journey.

- name: Source
  type: compound_input

2. Parse JSON Body

Logs are processed by the Parse JSON body node, which is an OTTL Transform node.

- name: Parse JSON body
  type: ottl_transform
  statements: set(attributes["parsed_body"], ParseJSON(Decode(body, "utf-8")))

The primary function of this node is to decode the message body as UTF-8, parse it into JSON, and store the result under the attributes["parsed_body"] attribute. By transforming raw JSON data into a more structured format, you enhance the ability to analyze and react to log content efficiently, improving overall observability.

3. Suppress Processor

The suppress processor node limit the number of similar messages within a specified time frame.

- name: Suppress Processor
  type: suppress
  interval: 30s
  key_field_paths:
  - attributes["parsed_body"]["ClientRequestHost"]
  - attributes["parsed_body"]["ClientASN"]
  - attributes["parsed_body"]["EdgeResponseStatus"]
  number_to_allow: 1

This node limits the visibility of repetitive log entries by checking specific fields (ClientRequestHost, ClientASN and EdgeResponseStatus) and allows only one log with the same property values within each 30-second interval.

4. Reserialize Body

Logs are then restructured by the Reserialize body node, another OTTL Transform node.

- name: Reserialize body
  type: ottl_transform
  statements: |-
    delete_matching_keys(attributes["parsed_body"], "ClientRequest")
    edx_delete_empty_values(attributes["parsed_body"], [], ["", "unknown"], ["deleteNull", "deleteZero"])
    set(body, EDXEncode(attributes["parsed_body"], "utf-8", true))
    delete_key(attributes, "parsed_body")    

This node performs the following functions:

1. The delete_matching_keys function removes all keys from the attributes["parsed_body"] map that match the regex pattern ClientRequest.
2. The edx_delete_empty_values custom function deletes keys with values that are empty from the attributes[“parsed_body”].
3. The set function sets the value of the body field. The value is obtained by encoding attributes["parsed_body"] into a byte array using the EDXEncode custom function with “utf-8” encoding.
4. The delete_key function removes the key parsed_body from the attributes map. This is useful for removing data that is no longer needed.

5. Final Output

The modified logs are routed from the Reserialize body node to the Processed compound output.

- name: Processed
  type: compound_output

Sample Input

{"Action":"log","ClientIP":"82.160.154.216","ClientRequestHost":"str.cribl.com","ClientRequestMethod":"GET","ClientRequestPath":"/str/js/gigyaUtils.js","ClientRequestQuery":"?20210320","Datetime":"2024-11-10T20:00:29Z","EdgeResponseStatus":200,"RayID":"7c546ce3ac6e0175","ClientASN":12322,"ClientASNDescription":"PROXAD","ClientCountry":"us","ClientIPClass":"noRecord","ClientRefererHost":"str.cribl.com","ClientRefererPath":"/str/signin","ClientRefererQuery":"?service=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&source=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&redirectAfterAccountLoginUrl=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&redirectAfterAccountCreationUrl=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&gauthHost=https%3A%2F%2Fstr.cribl.com%2Fstr&locale=us_us&id=gauth-widget&cssUrl=https%3A%2F%2Fstatic.criblcdn.com%2Fexpress%2Fgauth-windows-1.0.min.css&clientId=criblExpressWin&rememberMeShown=false&rememberMeChecked=false&createAccountShown=false&openCreateAccount=false&displayNameShown=false&consumeServiceTicket=true&initialFocus=true&embedWidget=true&generateExtraServiceTicket=false&generateTwoExtraServiceTickets=false&generateNoServiceTicket=false&globalOptInShown=false&globalOptInChecked=false&mobile=false&connectLegalTerms=true&showTermsOfUse=false&showPrivacyPolicy=false&showConnectLegalAge=false&locationPromptShown=true&showPassword=true&useCustomHeader=false&mfaRequired=false&performMFACheck=false&rememberMyBrowserShown=false&rememberMyBrowserChecked=false","ClientRefererScheme":"https","ClientRequestProtocol":"HTTP/2","ClientRequestScheme":"https","ClientRequestUserAgent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.208 Safari/537.36","Description":"","EdgeColoCode":"CDG","Kind":"firewall","MatchIndex":0,"Metadata":{"filter":"45fa5874f29344fea9ffc0f292ffdf34","type":"customer","js_detection":"MISSING"},"OriginResponseStatus":0,"OriginatorRayID":"00","Ref":"","RuleID":"9fc5732845ab49eea0ea37fd438326b2","Source":"firewallrules"}

{"Action":"log","ClientIP":"84.238.84.136","ClientRequestHost":"str.cribl.com","ClientRequestMethod":"GET","ClientRequestPath":"/str/js/consoleUtils.js","ClientRequestQuery":"?20210320","Datetime":"2024-11-10T20:01:16Z","EdgeResponseStatus":304,"RayID":"7c546e07c88ebe3d","ClientASN":33796,"ClientASNDescription":"BNAA-AS","ClientCountry":"dk","ClientIPClass":"noRecord","ClientRefererHost":"str.cribl.com","ClientRefererPath":"/str/login","ClientRefererQuery":"?service=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&source=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&redirectAfterAccountLoginUrl=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&redirectAfterAccountCreationUrl=https%3A%2F%2Fstr.cribl.com%2Fstr%2Fembed&gauthHost=https%3A%2F%2Fstr.cribl.com%2Fstr&locale=da&id=gauth-widget&cssUrl=https%3A%2F%2Fstatic.criblcdn.com%2Fcom.cribl.connect%2Fui%2Fcss%2Fgcm-str-theme-v1.7.css&clientId=criblConnectMobileAndroid&rememberMeShown=false&rememberMeChecked=false&createAccountShown=false&openCreateAccount=false&displayNameShown=false&consumeServiceTicket=true&initialFocus=true&embedWidget=true&socialEnabled=false&generateExtraServiceTicket=false&generateTwoExtraServiceTickets=false&generateNoServiceTicket=false&globalOptInShown=false&globalOptInChecked=false&mobile=true&connectLegalTerms=false&showTermsOfUse=false&showPrivacyPolicy=false&showConnectLegalAge=false&locationPromptShown=false&showPassword=true&useCustomHeader=false&mfaRequired=false&performMFACheck=false&rememberMyBrowserShown=false&rememberMyBrowserChecked=false&prepopUsername=stinneyou%40hotmail.com","ClientRefererScheme":"https","ClientRequestProtocol":"HTTP/2","ClientRequestScheme":"https","ClientRequestUserAgent":"Mozilla/5.0 (Linux; Android 11; GM1823 Build/RKQ1.201022.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/112.0.5615.135 Mobile Safari/537.36","Description":"","EdgeColoCode":"CPH","Kind":"firewall","MatchIndex":0,"Metadata":{"filter":"45fa5874f29344fea9ffc0f292ffdf34","type":"customer","js_detection":"MISSING"},"OriginResponseStatus":0,"OriginatorRayID":"00","Ref":"","RuleID":"9fc5732845ab49eea0ea37fd438326b2","Source":"firewallrules"}