Akamai JSON Pack

This is a pack that allows for processing of Akamai JSON logs. It includes masking, parsing the json, transforming and dropping excess fields and extracting a log body from the json.

  4 minute read  

Edge Delta Pipeline Pack for Akamai

Overview

The Akamai pipeline pack ensures appropriate processing of Akamai log data. This pipeline manages the ingestion process by connecting structured log data from Akamai to Edge Delta and providing crucial insights into web traffic and security events. With this pipeline, you can monitor, detect anomalies, and generate metrics from Akamai logs.

Pack Description

1. Data Ingestion

The data flow starts with the compound_input node. This node serves as the entry point into the pipeline, where it begins processing the incoming Akamai logs.

2. Drop Empty Fields

Next, logs flow into the drop-empty node, which is a Mask node. This node removes unimportant fields from logs by masking specific patterns.

  - name: drop-empty
    type: mask
    pattern: \,\"\w+\"\:\"\-\"\,
    mask: ','

The pattern parameter specifies a regex pattern for identifying parts of the log to mask. In this case, the regex looks for commas followed by empty fields. The mask parameter defines what to replace these patterns with, which is a comma. This operation reduces log size by removing fields that contain no valuable data, making it easier to process essential information later.

3. Drop Empty End Fields

The drop-empty-end node is another Mask node focusing on the end of JSON objects in logs.

  - name: drop-empty-end
    type: mask
    pattern: \,\"\w+\"\:\"\-\"\}
    mask: '}'

This node uses a similar pattern to drop-empty but targets empty fields at the end of JSON objects. By ensuring such fields are removed or masked at the end of objects, you avoid processing unnecessary data in later stages.

4. Parse JSON Attributes

Logs then flow to the parse_json_attributes node, which is a parse_json_attributes node. This node parses JSON formatted logs and converts their structure into attributes.

The node processes log entries by extracting fields outlined in JSON format within the log body, transforming these into more accessible attributes for downstream processing.

5. Drop Unimportant Fields

Subsequently, logs move to the drop-unimportant-fields node, a Log Transform node, which deletes less significant attributes from logs.

  - name: drop-unimportant-fields
    type: log_transform
    transformations:
      - field_path: item.attributes.cacheable
        operation: delete
      - field_path: item.attributes.tlsVersion
        operation: delete
      - field_path: item.attributes.version
        operation: delete

Here, the transformations parameter includes a list of operations to delete specified attributes (cacheable, tlsVersion, and version). Each field_path indicates the attribute to be targeted, and the operation specifies that it should be deleted. This enhances focus on only relevant data by removing fields that are not crucial for further log analysis.

6. Extract JSON Field

Finally, logs are processed by the extract_path node, which is an Extract JSON Field node. This node extracts specific field data from JSON logs.

  - name: extract_path
    type: extract_json_field
    field_path: reqPath
    keep_log_if_failed: true

The field_path parameter specifies the JSON path to extract (reqPath). By setting keep_log_if_failed to true, logs still flow through even if extraction fails, ensuring no data loss. This operation provides precise extraction, important for further drills into specific details within complex logs.

7. Processed Logs Output

The processed-logs node, a compound_output node, then captures and outputs the processed logs, completing the data flow. This outputs cleaned and refined logs for archival or further analysis, facilitating the efficient use of data.

Example Input

{
  "version": 1,
  "streamId": "54321",
  "cp": "654321",
  "reqId": "abcd1234",
  "reqTimeSec": "1573840000",
  "bytes": "4995",
  "cliIP": "192.0.2.1",
  "statusCode": "206",
  "proto": "HTTPS",
  "reqHost": "example.hostname.net",
  "reqMethod": "GET",
  "reqPath": "/newpath/anotherpath/resource.ext",
  "reqPort": "443",
  "rspContentLen": "4000",
  "rspContentType": "application/json",
  "UA": "Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%29",
  "tlsOverheadTimeMSec": "5",
  "tlsVersion": "TLSv1.2",
  "objSize": "300",
  "uncompressedSize": "350",
  "overheadBytes": "100",
  "totalBytes": "0",
  "queryStr": "param=newvalue",
  "breadcrumbs": "//BC/%5Ba=203.0.113.1,c=j,k=1,l=0%5D",
  "accLang": "en-GB",
  "cookie": "new-cookie-content",
  "range": "10000-20000",
  "referer": "https%3A%2F%2Fnew.referrer.net%2Fdocs%2FWeb%2Fexample",
  "xForwardedFor": "203.0.113.2",
  "maxAgeSec": "7200",
  "reqEndTimeMSec": "10",
  "errorCode": "ERR_NETWORK_CHANGED|acl_fwd",
  "turnAroundTimeMSec": "15",
  "transferTimeMSec": "150",
  "dnsLookupTimeMSec": "30",
  "lastByte": "5",
  "edgeIP": "198.51.100.2",
  "country": "US",
  "state": "California",
  "city": "San Francisco",
  "serverCountry": "JP",
  "billingRegion": "9",
  "cacheStatus": "2",
  "securityRules": "ULnR_98765|1230000:1230001:1230005:4560006:BOT-ANOMALY-BODY|",
  "ewUsageInfo": "//1234/1.0/1/-/0/2/#2,1\//1234/1.0/2/-/0/1/#1,0\//1234/2.0/3/-/1/3/#0,1",
  "ewExecutionInfo": "a:1234:5:105:107:105:n:::12345:404|B:1234:2:0:1:0:n:::6789:403|S:1234:10:50:50:1:n:::14789:201",
  "customField": "new-custom-value"
}

Example Output

{
  "_type": "log",
  "attributes": {
    "UA": "Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%29",
    "accLang": "en-GB",
    "billingRegion": "9",
    "breadcrumbs": "//BC/%5Ba=203.0.113.1,c=j,k=1,l=0%5D",
    "bytes": "4995",
    "cacheStatus": "2",
    "city": "San Francisco",
    "cliIP": "192.0.2.1",
    "cookie": "new-cookie-content",
    "country": "US",
    "cp": "654321",
    "customField": "new-custom-value",
    "dnsLookupTimeMSec": "30",
    "edgeIP": "198.51.100.2",
    "errorCode": "ERR_NETWORK_CHANGED|acl_fwd",
    "ewExecutionInfo": "a:1234:5:105:107:105:n:::12345:404|B:1234:2:0:1:0:n:::6789:403|S:1234:10:50:50:1:n:::14789:201",
    "ewUsageInfo": "//1234/1.0/1/-/0/2/#2,1\//1234/1.0/2/-/0/1/#1,0\//1234/2.0/3/-/1/3/#0,1",
    "lastByte": "5",
    "maxAgeSec": "7200",
    "objSize": "300",
    "overheadBytes": "100",
    "proto": "HTTPS",
    "queryStr": "param=newvalue",
    "range": "10000-20000",
    "referer": "https%3A%2F%2Fnew.referrer.net%2Fdocs%2FWeb%2Fexample",
    "reqEndTimeMSec": "10",
    "reqHost": "example.hostname.net",
    "reqId": "abcd1234",
    "reqMethod": "GET",
    "reqPath": "/newpath/anotherpath/resource.ext",
    "reqPort": "443",
    "reqTimeSec": "1573840000",
    "rspContentLen": "4000",
    "rspContentType": "application/json",
    "securityRules": "ULnR_98765|1230000:1230001:1230005:4560006:BOT-ANOMALY-BODY|",
    "serverCountry": "JP",
    "state": "California",
    "statusCode": "206",
    "streamId": "54321",
    "tlsOverheadTimeMSec": "5",
    "totalBytes": "0",
    "transferTimeMSec": "150",
    "turnAroundTimeMSec": "15",
    "uncompressedSize": "350",
    "xForwardedFor": "203.0.113.2"
  },
  "body": "/newpath/anotherpath/resource.ext",
  "resource": {
    "ed.conf.id": "12345678987654321",
    "ed.org.id": "98765432123456789",
    "ed.tag": "akamaitest",
    "host.ip": "10.0.0.1",
    "host.name": "ED_TEST",
    "service.name": "",
    "src_type": "memory_input"
  },
  "timestamp": 1722492920223
}

In the output log:

  • Nested JSON attributes are converted to standalone top-level attributes.
  • Unimportant fields are deleted.
  • The reqPath field is correctly extracted and stored in the body field.

Sample Input

{"version":1,"streamId":"12345","cp":"123456","reqId":"1239f220","reqTimeSec":"1573840000","bytes":"4995","cliIP":"128.147.28.68","statusCode":"206","proto":"HTTPS","reqHost":"test.hostname.net","reqMethod":"GET","reqPath":"/path1/path2/file.ext","reqPort":"443","rspContentLen":"5000","rspContentType":"text/html","UA":"Mozilla%2F5.0+%28Macintosh%3B+Intel+Mac+OS+X+10_14_3%29","tlsOverheadTimeMSec":"0","tlsVersion":"TLSv1","objSize":"484","uncompressedSize":"484","overheadBytes":"232","totalBytes":"0","queryStr":"param=value","breadcrumbs":"//BC/%5Ba=23.33.41.20,c=g,k=0,l=1%5D","accLang":"en-US","cookie":"cookie-content","range":"37334-42356","referer":"https%3A%2F%2Ftest.referrer.net%2Fen-US%2Fdocs%2FWeb%2Ftest","xForwardedFor":"8.47.28.38","maxAgeSec":"3600","reqEndTimeMSec":"3","errorCode":"ERR_ACCESS_DENIED|fwd_acl","turnAroundTimeMSec":"11","transferTimeMSec":"125","dnsLookupTimeMSec":"50","lastByte":"1","edgeIP":"23.50.51.173","country":"IN","state":"Virginia","city":"HERNDON","serverCountry":"SG","billingRegion":"8","cacheStatus":"1","securityRules":"ULnR_28976|3900000:3900001:3900005:3900006:BOT-ANOMALY-HEADER|","ewUsageInfo":"//4380/4.0/1/-/0/4/#1,2\\//4380/4.0/4/-/0/4/#0,0\\//4380/4.0/5/-/1/1/#0,0","ewExecutionInfo":"c:4380:7:161:162:161:n:::12473:200|C:4380:3:0:4:0:n:::6967:200|R:4380:20:99:99:1:n:::35982:200","customField":"any-custom-value"}

{"version":1,"streamId":"98765","cp":"765432","reqId":"zxc9876vbn","reqTimeSec":"1697653200","bytes":"20456","cliIP":"192.168.1.14","statusCode":"304","proto":"HTTP/1.1","reqHost":"cdn.example.org","reqMethod":"POST","reqPath":"/api/v1/upload","reqPort":"80","rspContentLen":"0","rspContentType":"application/json","UA":"curl/7.64.1","tlsOverheadTimeMSec":"5","tlsVersion":"TLSv1.2","objSize":"20000","uncompressedSize":"20000","overheadBytes":"56","totalBytes":"20456","queryStr":"apiKey=abc123&format=json","breadcrumbs":"//BC/[a=192.168.1.14,c=g,k=2,l=2]","accLang":"fr-FR,fr;q=0.8,en-US;q=0.5","cookie":"sessionToken=123xyz456;userPref=darkmode","range":"","referer":"https://www.example.org/dashboard","xForwardedFor":"172.16.254.1","maxAgeSec":"0","reqEndTimeMSec":"15","errorCode":"","turnAroundTimeMSec":"20","transferTimeMSec":"80","dnsLookupTimeMSec":"10","lastByte":"1","edgeIP":"198.51.100.101","country":"FR","state":"Île-de-France","city":"Paris","serverCountry":"DE","billingRegion":"3","cacheStatus":"MISS","securityRules":"RULE5678|4500000:BOT-SUSPECTED|","ewUsageInfo":"//1234/5.1/2/-/0/3/#1,0\\//1234/5.1/5/-/1/2/#0,1","ewExecutionInfo":"c:1234:8:100:101:102:n:::9876:304|C:1234:4:0:5:0:n:::1234:304|R:1234:19:88:88:2:n:::54321:304","customField":"custom-unique-value-789"}