Edge Delta Pipeline Pack for Akamai

Pre-designed pipeline configuration for Akamai logs.

6 minute read

Overview

The Akamai pipeline pack ensures appropriate processing of Akamai log data. This pipeline manages the ingestion process by connecting structured log data from Akamai to Edge Delta and providing crucial insights into web traffic and security events. With this pipeline, you can monitor, detect anomalies, and generate metrics from Akamai logs.

Pack Integration Process

Pipeline packs are best implemented as Compound Nodes for re-use and change control.

To create a new compound node containing the pack:

1. Create and deploy a new fleet:

Deploy a temporary fleet. You will use this fleet to create the compound node and to test the pack.

Click Pipelines.
Click New Fleet.
Select Edge Fleet.
Optionally, expand Advanced Fleet Configuration and select a pipeline with a configuration you want to duplicate.
Click Continue.
Select the appropriate template and click Continue.
Specify a name to identify the Fleet.
Click Generate Config.
Execute the installation commands, they include the unique ID for the Fleet.
Expand the namespaces and select the input sources you want to monitor.
Select the Destination Outputs you want to send processed data to, such as the Edge Delta Observability Platform.
Click Continue.
Click View Dashboard.

2. Replace the Fleet Yaml

Open the yaml editor for the newly created fleet. Replace the links and nodes sections with the following YAML:

nodes:
- name: drop-empty
  type: mask
  pattern: ',\s*"[^"]+"\s*:\s*"-"\s*,'
  mask: ','
- name: drop-empty-end
  type: mask
  pattern: \,\"\w+\"\:\"\-\"\}
  mask: '}'
- name: parse_json_attributes
  type: parse_json_attributes
- name: drop-unimportant-fields
  type: log_transform
  transformations:
  - field_path: item.attributes.cacheable
    operation: delete
  - field_path: item.attributes.tlsVersion
    operation: delete
  - field_path: item.attributes.version
    operation: delete
- name: extract_path
  type: extract_json_field
  field_path: reqPath
  keep_log_if_failed: true
links:
- from: drop-empty
  to: drop-empty-end
- from: parse_json_attributes
  to: drop-unimportant-fields
- from: drop-empty-end
  to: parse_json_attributes
- from: drop-unimportant-fields
  to: extract_path

To edit a v3 configuration using YAML:

Click Pipelines and select the fleet you want to edit.
Click View/Edit Pipeline.
Click Edit YAML.

The Edit Configuration dialog box opens. You can edit the configuration and view the change diff.

Click Done.

3. Create a Compound Node

Click Edit Mode
Select the nodes using one of these methods:

hover over each node and check the selection checkbox,
hold Command select each node, or
hold Shift and drag a selection box over the nodes.

Select Create Compound Node.

The nodes are grouped in a single compound node and the new compound node is added to the compound node library. Now you switch to the compound node library editor.

Click Pipeline and select the Compound Nodes tab.
Click Confirm.

4. Wire up the Compound Node

Click the new Compound Node.
Specify a name and description for the pack.
Click Add Destination.
Name the destination node Failures and click Save Changes.
Connect the Compound_input node to the drop_empty node.
Connect the default path on the extract_path node to the compound_output node.
Connect the Failures destination node to the Failure path on the extract_path node and also to the Failure path on the Parse_JSON_Attributes node.
Click Review Changes.
Click Save.
Click Save.

Akamai Pack Contents

The Akamai Compound Node consists of a series of nodes, each of which will be discussed in turn:

Logs flow through each node in the following order:

1. drop_empty

This Mask node is useful at the beginning of the pipeline to handle empty fields in the logs. The regular expression is designed to identify fields with “-” values and mask these with a comma. This step helps to reduce the size of logs by eliminating irrelevant data.

2. drop-empty-end

This Mask node targets trailing empty fields at the end of the JSON object. By identifying these trailing empty fields and replacing them with a closing brace, the node ensures the JSON objects are cleanly terminated. This consistency is vital for accurate parsing in subsequent processing steps.

3. parse_json_attributes

The Parse JSON node converts nested JSON attributes from the item[“body”] field into standalone attributes. This transformation simplifies access to individual fields within the logs for further processing and querying. This step is particularly important for Akamai logs, which often contain complex nested JSON data that needs to be flattened for easier analysis.

4. drop-unimportant-fields

This log_transform node reduce data size and noise by removing fields deemed unnecessary for analysis. Fields like item.attributes.cacheable, item.attributes.tlsVersion, and item.attributes.version are deleted. Removing these fields helps focus on the most relevant data, conserving resources and optimizing performance.

5. extract_path

The Extract JSON Field node focuses on extracting the reqPath field from the JSON logs. This field is often crucial in Akamai logs as it can indicate which resources were requested. Extracting this specific field aids in deeper analysis such as web traffic pattern recognition or security event tracking. Importantly, logs where this extraction fails are still retained (keep_log_if_failed: true), ensuring no potentially valuable data is lost even if some logs do not contain the reqPath field.

Testing the Pack

To test the pack, add it to a fleet and open it in editing mode. You can add it to the original pipeline that you used to create it in a previous step.

Click Pipelines.
Select a test fleet.
Click View/Edit Pipeline.
Click Edit Mode.
Click Add Processor.
Expand Compound Nodes and click the Akamai pack (or whichever name you specified in a previous step)
Paste a test log into the Samples box (see Example Input).
Click Process Samples.
Evaluate the Outbound data field to ensure the sample was processed as expected (see Example Output). Also ensure that the data is on the compound_output path, and not the Failures path.

Example Input

{
  "version": 1,
  "streamId": "54321",
  "cp": "654321",
  "reqId": "abcd1234",
  "reqTimeSec": "1573840000",
  "bytes": "4995",
  "cliIP": "192.0.2.1",
  "statusCode": "206",
  "proto": "HTTPS",
  "reqHost": "example.hostname.net",
  "reqMethod": "GET",
  "reqPath": "/newpath/anotherpath/resource.ext",
  "reqPort": "443",
  "rspContentLen": "4000",
  "rspContentType": "application/json",
  "UA": "Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%29",
  "tlsOverheadTimeMSec": "5",
  "tlsVersion": "TLSv1.2",
  "objSize": "300",
  "uncompressedSize": "350",
  "overheadBytes": "100",
  "totalBytes": "0",
  "queryStr": "param=newvalue",
  "breadcrumbs": "//BC/%5Ba=203.0.113.1,c=j,k=1,l=0%5D",
  "accLang": "en-GB",
  "cookie": "new-cookie-content",
  "range": "10000-20000",
  "referer": "https%3A%2F%2Fnew.referrer.net%2Fdocs%2FWeb%2Fexample",
  "xForwardedFor": "203.0.113.2",
  "maxAgeSec": "7200",
  "reqEndTimeMSec": "10",
  "errorCode": "ERR_NETWORK_CHANGED|acl_fwd",
  "turnAroundTimeMSec": "15",
  "transferTimeMSec": "150",
  "dnsLookupTimeMSec": "30",
  "lastByte": "5",
  "edgeIP": "198.51.100.2",
  "country": "US",
  "state": "California",
  "city": "San Francisco",
  "serverCountry": "JP",
  "billingRegion": "9",
  "cacheStatus": "2",
  "securityRules": "ULnR_98765|1230000:1230001:1230005:4560006:BOT-ANOMALY-BODY|",
  "ewUsageInfo": "//1234/1.0/1/-/0/2/#2,1\\//1234/1.0/2/-/0/1/#1,0\\//1234/2.0/3/-/1/3/#0,1",
  "ewExecutionInfo": "a:1234:5:105:107:105:n:::12345:404|B:1234:2:0:1:0:n:::6789:403|S:1234:10:50:50:1:n:::14789:201",
  "customField": "new-custom-value"
}

Example Output

{
  "_type": "log",
  "attributes": {
    "UA": "Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%29",
    "accLang": "en-GB",
    "billingRegion": "9",
    "breadcrumbs": "//BC/%5Ba=203.0.113.1,c=j,k=1,l=0%5D",
    "bytes": "4995",
    "cacheStatus": "2",
    "city": "San Francisco",
    "cliIP": "192.0.2.1",
    "cookie": "new-cookie-content",
    "country": "US",
    "cp": "654321",
    "customField": "new-custom-value",
    "dnsLookupTimeMSec": "30",
    "edgeIP": "198.51.100.2",
    "errorCode": "ERR_NETWORK_CHANGED|acl_fwd",
    "ewExecutionInfo": "a:1234:5:105:107:105:n:::12345:404|B:1234:2:0:1:0:n:::6789:403|S:1234:10:50:50:1:n:::14789:201",
    "ewUsageInfo": "//1234/1.0/1/-/0/2/#2,1\\//1234/1.0/2/-/0/1/#1,0\\//1234/2.0/3/-/1/3/#0,1",
    "lastByte": "5",
    "maxAgeSec": "7200",
    "objSize": "300",
    "overheadBytes": "100",
    "proto": "HTTPS",
    "queryStr": "param=newvalue",
    "range": "10000-20000",
    "referer": "https%3A%2F%2Fnew.referrer.net%2Fdocs%2FWeb%2Fexample",
    "reqEndTimeMSec": "10",
    "reqHost": "example.hostname.net",
    "reqId": "abcd1234",
    "reqMethod": "GET",
    "reqPath": "/newpath/anotherpath/resource.ext",
    "reqPort": "443",
    "reqTimeSec": "1573840000",
    "rspContentLen": "4000",
    "rspContentType": "application/json",
    "securityRules": "ULnR_98765|1230000:1230001:1230005:4560006:BOT-ANOMALY-BODY|",
    "serverCountry": "JP",
    "state": "California",
    "statusCode": "206",
    "streamId": "54321",
    "tlsOverheadTimeMSec": "5",
    "totalBytes": "0",
    "transferTimeMSec": "150",
    "turnAroundTimeMSec": "15",
    "uncompressedSize": "350",
    "xForwardedFor": "203.0.113.2"
  },
  "body": "/newpath/anotherpath/resource.ext",
  "resource": {
    "ed.conf.id": "12345678987654321",
    "ed.org.id": "98765432123456789",
    "ed.tag": "akamaitest",
    "host.ip": "10.0.0.1",
    "host.name": "ED_TEST",
    "service.name": "",
    "src_type": "memory_input"
  },
  "timestamp": 1722492920223
}

In the output log:

Nested JSON attributes are converted to standalone top-level attributes.
Unimportant fields are deleted.
The reqPath field is correctly extracted and stored in the body field.